# Spack install Firedrake on ARCHER2
###### tags: `spack` `archer2`
We will work with the in a directory `/work/[budget code]/[budget code]/[$USER]/workspace`, where `[budget code]` is replaced by your project code and `[$USER]` is your username.
## Prerequisites
You should familiarise yourself with as much of Spack's functionality as possible. To achieve the same flexibility as we have on desktop installs the Firedrake Spack install leverages a lot of Spack's features. More information about Spack can be found on their [website](https://spack.readthedocs.io/en/latest/index.html).
## Spack
Start in a directory on the WORK filesystem, not HOME as it is not accessible from the compute nodes. You may wish to set `SPACK_USER_CONFIG_PATH` which defaults to `~/.spack` to change where spack looks for user level configuration files. Changing `SPACK_USER_CONFIG_PATH` isn't necessary if everything is correctly configured within the spack environment.
### Environment
We recommend using the GNU toolchain and GCC compilers at version 10.2.0, see [additional notes](#Additional-notes) for more info.
```bash
git clone -c feature.manyFiles=true git@github.com:spack/spack.git
module purge
module load load-epcc-module
# Load programming environment and compiler
module load PrgEnv-gnu/8.1.0
module swap gcc gcc/10.2.0
module swap cray-mpich cray-mpich/8.1.9
module swap cray-dsmml cray-dsmml/0.2.1
module swap cray-libsci cray-libsci/21.08.1.2
module load xpmem
module load cmake/3.21.3
module load cray-python/3.9.4.1
. spack/share/spack/setup-env.sh
```
If you want to use a different BLAS and LAPACK and Scalapack library you don't need to load cray-libsci and you should run `module unload cray-libsci`. For instance the AMD optimised CPU libraries (AOCL) could be loaded instead `module load aocl/3.1`
If you _really_ want to try the other compilers replace the two isolated lines under `# Load programming environment and compiler` with
```
module load PrgEnv-aocc/8.1.0
module swap aocc aocc/3.0.0
# or
module load PrgEnv-cray/8.1.0
module swap cce cce/12.0.3
```
for the AMD or Cray compilers respectively.
### Configure spack
1. Populate `$SPACK_USER_CONFIG_PATH/cray/compilers.yaml` with the following YAML. Note that this contains most of the available compilers, you only need to configure for the compilers you will use.
<details>
<summary>
<code>$SPACK_USER_CONFIG_PATH/cray/compilers.yaml</code>
</summary>
```yaml
compilers:
- compiler:
spec: aocc@2.2.0
paths:
cc: cc
cxx: CC
f77: ftn
fc: ftn
flags:
cflags: null
cxxflags: null
fflags: null
operating_system: sles15
target: any
modules:
- PrgEnv-aocc
- aocc/2.2.0
environment: {}
extra_rpaths: []
- compiler:
spec: aocc@2.2.0.1
paths:
cc: cc
cxx: CC
f77: ftn
fc: ftn
flags:
cflags: null
cxxflags: null
fflags: null
operating_system: sles15
target: any
modules:
- PrgEnv-aocc
- aocc/2.2.0.1
environment: {}
extra_rpaths: []
- compiler:
spec: aocc@3.0.0
paths:
cc: cc
cxx: CC
f77: ftn
fc: ftn
flags:
cflags: -Wno-unused-command-line-argument -mllvm -eliminate-similar-expr=false
cxxflags: -Wno-unused-command-line-argument -mllvm -eliminate-similar-expr=false
fflags: -Wno-unused-command-line-argument -mllvm -eliminate-similar-expr=false
ldflags: -Wl,-rpath,/opt/cray/libfabric/1.11.0.4.71/lib64 -L/opt/cray/libfabric/1.11.0.4.71/lib64 -lfabric
operating_system: sles15
target: any
modules:
- PrgEnv-aocc
- aocc/3.0.0
environment: {}
extra_rpaths: []
- compiler:
spec: cce@11.0.4
paths:
cc: cc
cxx: CC
f77: ftn
fc: ftn
flags: {}
operating_system: sles15
target: any
modules:
- PrgEnv-cray
- cce/11.0.4
environment: {}
extra_rpaths: []
- compiler:
spec: cce@12.0.3
paths:
cc: cc
cxx: CC
f77: ftn
fc: ftn
flags: {}
operating_system: sles15
target: any
modules:
- PrgEnv-cray
- cce/12.0.3
environment: {}
extra_rpaths: []
- compiler:
spec: gcc@9.3.0
paths:
cc: cc
cxx: CC
f77: ftn
fc: ftn
flags: {}
operating_system: sles15
target: any
modules:
- PrgEnv-gnu
- gcc/9.3.0
environment: {}
extra_rpaths: []
- compiler:
spec: gcc@10.2.0
paths:
cc: cc
cxx: CC
f77: ftn
fc: ftn
flags: {}
operating_system: sles15
target: any
modules:
- PrgEnv-gnu
- gcc/10.2.0
environment: {}
extra_rpaths: []
- compiler:
spec: gcc@10.3.0
paths:
cc: cc
cxx: CC
f77: ftn
fc: ftn
flags: {}
operating_system: sles15
target: any
modules:
- PrgEnv-gnu
- gcc/10.3.0
environment: {}
extra_rpaths: []
- compiler:
spec: gcc@11.2.0
paths:
cc: cc
cxx: CC
f77: ftn
fc: ftn
flags: {}
operating_system: sles15
target: any
modules:
- PrgEnv-gnu
- gcc/11.2.0
environment: {}
extra_rpaths: []
- compiler:
spec: rocmcc@2.14-nosmp
paths:
cc: cc
cxx: CC
f77: ftn
fc: ftn
flags: {}
operating_system: sles15
target: any
modules:
- PrgEnv-amd
- rocmcc/2.14-nosmp
environment: {}
extra_rpaths: []
- compiler:
spec: rocmcc@2.14
paths:
cc: cc
cxx: CC
f77: ftn
fc: ftn
flags: {}
operating_system: sles15
target: any
modules:
- PrgEnv-amd
- rocmcc/2.14
environment: {}
extra_rpaths: []
```
</details>
2. Populate `$SPACK_USER_CONFIG_PATH/packages.yaml` with the following
<details>
<summary>
<code>$SPACK_USER_CONFIG_PATH/packages.yaml</code>
</summary>
```yaml
packages:
all:
providers:
mpi: [cray-mpich]
cray-mpich:
externals:
- spec: cray-mpich@8.1.9%gcc@10.2.0
modules:
- PrgEnv-gnu
- gcc/10.2.0
- cray-mpich/8.1.9
- spec: cray-mpich@8.1.6%aocc@3.0.0
# Pretend mpich is version 8.1.6 due to the following bug in spack
# https://github.com/spack/spack/issues/29459
# Only with aocc compiler
modules:
- PrgEnv-aocc
- aocc/3.0.0
- cray-mpich/8.1.9
prefix: /opt/cray/pe/mpich/8.1.9/ofi/aocc/3.0
- spec: cray-mpich@8.1.9%cce@12.0.3
modules:
- PrgEnv-cray
- cce/12.0.3
- cray-mpich/8.1.9
buildable: False
cray-libsci:
externals:
- spec: cray-libsci@21.04.1.1
modules:
- cray-libsci/21.04.1.1
- spec: cray-libsci@21.08.1.2
modules:
- cray-libsci/21.08.1.2
environment:
append_path:
- PE_PKGCONFIG_PRODUCTS: PE_MPICH
buildable: False
amdblis:
externals:
- spec: amdblis@3.1
modules:
- aocl/3.1
buildable: False
amdfftw:
externals:
- spec: amdfftw@3.1
modules:
- aocl/3.1
buildable: False
amdlibflame:
externals:
- spec: amdlibflame@3.1
modules:
- aocl/3.1
buildable: False
amdlibm:
externals:
- spec: amdlibm@3.1
modules:
- aocl/3.1
buildable: False
amdscalapack:
externals:
- spec: amdscalapack@3.1
modules:
- aocl/3.1
buildable: False
aocl-sparse:
externals:
- spec: aocl-sparse@3.1
modules:
- aocl/3.1
buildable: False
python:
externals:
- spec: python@3.9.4.1
modules:
- cray-python/3.9.4.1
- spec: python@3.8.5.0
modules:
- cray-python/3.8.5.0
buildable: False
flex:
externals:
- spec: flex@2.6.4+lex
prefix: /usr
# System texinfo causes issues!
# texinfo:
# externals:
# - spec: texinfo@6.5
# prefix: /usr
automake:
externals:
- spec: automake@1.15.1
prefix: /usr
binutils:
externals:
- spec: binutils@2.35.1
prefix: /usr
bison:
externals:
- spec: bison@3.0.4
prefix: /usr
libtool:
externals:
- spec: libtool@2.4.6
prefix: /usr
m4:
externals:
- spec: m4@1.4.18
prefix: /usr
openssh:
externals:
- spec: openssh@7.9p1
prefix: /usr
gawk:
externals:
- spec: gawk@4.2.1
prefix: /usr
pkg-config:
externals:
- spec: pkg-config@0.29.2
prefix: /usr
buildable: False
pkgconf:
externals:
- spec: pkg-config@0.29.2
prefix: /usr
buildable: False
openssl:
externals:
- spec: openssl@1.1.0i-fips
prefix: /usr
groff:
externals:
- spec: groff@1.22.3
prefix: /usr
autoconf:
externals:
- spec: autoconf@2.69
prefix: /usr
findutils:
externals:
- spec: findutils@4.6.0
prefix: /usr
git:
externals:
- spec: git@2.26.2~tcltk
prefix: /usr
tar:
externals:
- spec: tar@1.30
prefix: /usr
subversion:
externals:
- spec: subversion@1.10.0
prefix: /usr
cmake:
externals:
- spec: cmake@3.10.2
prefix: /usr
- spec: cmake@3.21.3
prefix: /work/y07/shared/utils/core/cmake/3.21.3
gmake:
externals:
- spec: gmake@4.2.1
prefix: /usr
diffutils:
externals:
- spec: diffutils@3.6
prefix: /usr
```
</details>
## Firedrake
### Installation
1. Clone the repo:
```bash
git clone https://github.com/firedrakeproject/firedrake-spack.git
# or
git clone git@github.com:firedrakeproject/firedrake-spack.git
```
1. Add the repository to spack `spack repo add <repo directory> `
1. Create an spack environment `spack env create -d ./firedrake`
1. Activate that environment `spack env activate -p ./firedrake`
1. To avoid a bunch of errors add a whole bunch of packages to the development package list:
```
spack develop py-firedrake@develop
spack develop libsupermesh@develop
spack develop petsc@develop
spack develop chaco@petsc
spack develop py-fiat@develop
spack develop py-finat@develop
spack develop py-islpy@develop
spack develop py-petsc4py@develop
spack develop py-pyadjoint@develop
spack develop py-pyop2@develop
spack develop py-coffee@develop
spack develop py-loopy@develop
spack develop py-cgen@develop
spack develop py-codepy@develop
spack develop py-genpy@develop
spack develop py-tsfc@develop
spack develop py-ufl@develop
```
1. To work around the fact that `/home` is not accessible from compute nodes the location of the firedrake repo needs to be set inside `$SPACK_ENV/spack.yaml`:
```yaml
spack:
repos:
- /work/[budget code]/[budget code]/[$USER]/workspace/firedrake-spack
```
Where the path should point to the repo cloned in step 1.
1. Then custom configure the packages inside `$SPACK_ENV/spack.yaml`:
```yaml
spack:
include:
- packages.yaml
```
1. And create `$SPACK_ENV/packages.yaml` with contents:
```yaml
packages:
all:
compiler: [gcc@10.2.0]
providers:
mpi: [cray-mpich]
blas: [cray-libsci]
lapack: [cray-libsci]
scalapack: [cray-libsci]
netlib-scalapack:
externals:
# Pretend cray-libsci is netlib-scalapack
- spec: netlib-scalapack@21.08.1.2
modules:
- cray-libsci/21.08.1.2
buildable: False
py-numpy:
externals:
- spec: py-numpy@1.20.1
modules:
- cray-libsci/21.08.1.2
- cray-python/3.9.4.1
prefix: /opt/cray/pe/python/3.9.4.1/
buildable: False
py-scipy:
externals:
- spec: py-scipy@1.6.2
modules:
- cray-libsci/21.08.1.2
- cray-python/3.9.4.1
prefix: /opt/cray/pe/python/3.9.4.1/
buildable: False
```
If using AMD or Cray compilers replace the compiler above (`[gcc@10.2.0]`) with `[aocc@3.0.0]` or `[cce@12.0.3]`.
1. Add Firedrake to the spec for the environment
```
spack add py-firedrake@develop \
%gcc@10.2.0 \
^python@3.9.4.1 \
^cray-mpich@8.1.9%gcc@10.2.0 \
^cray-libsci@21.08.1.2 \
^firedrake.petsc@develop+fftw \
cflags=\"-O3 -march=native -mtune=native\" \
cxxflags=\"-O3 -march=native -mtune=native\" \
fflags=\"-O3 -march=native -mtune=native -ffree-line-length-512\"
```
1. Concretize (and log) `spack concretize -f 2>&1 | tee $SPACK_ENV/conc.log`
* Splitting the concretize and install steps into 2 distinct parts allows you to review what is being installed before the lengthy installation process begins.
1. Install (and log) `spack install --fail-fast 2>&1 | tee $SPACK_ENV/spack-firedrake-install.log`
### Testing
Testing must be run on a compute node. An interactive session can be started using
```bash
srun --nodes=1 --exclusive --time=00:20:00 \
--partition=standard --qos=short --reservation=shortqos \
--account=[budget code] --pty /bin/bash
```
Alternatively submit a jobscript, see [additional notes](#Additional-notes).
1. Test you can import Firedrake by running `python -c "from firedrake import *"`
- If this fails, before trying anything else, deactivate the environment with `spack env deactivate` and reactivate with `spack env activate -p ./firedrake` (as above) and try running `python -c "from firedrake import *"` again. This appears to be a shortcoming of spack.
1. Run the basic functionality tests:
```bash
cd $SPACK_ENV/py-firedrake
pytest tests/regression/ -m "not parallel" -k "poisson_strong or stokes_mini or dg_advection"
```
- When running the environment on the compute nodes the variable `$HOME` needs to be set to something like `/work/_projectcode_/_projectcode_/_username_` or `/tmp` as many dependencies expect to find configuration in `$HOME`.
- We don't run parallel tests due to an issue with the current parallel pytest hook, this is in the process of being fixed.
1. Run the full test suite:
```bash
cd $SPACK_ENV/src/firedrake
pytest tests -m "not parallel"
```
### General issues encountered
This section contains development notes, and is only included for future development and general interest. Skip this section to read about configuring for AMD or Cray compilers.
It's almost certainly better to do everything in this section on a per env basis. If this is done globally in `~/.spack/packages.yaml` it will not be possible to install other versions of these packages.
- MUMPS insists on building with `netlib-scalapack`. To use `cray-libsci` we can create an external packages called `netlib-scalapack` that loads the `cray-libsci` modules and add it to `$SPACK_ENV/packages.yaml`. **This may not be a good idea! We included what is below in step 8 above as it seems to work YMMV.**
```yaml
packages:
netlib-scalapack:
externals:
# Pretend cray-libsci is netlib-scalapack
- spec: netlib-scalapack@21.08.1.2
modules:
- cray-libsci/21.08.1.2
buildable: False
```
- It's not really possible to build `numpy` and `scipy` using the `cray-libsci` library, that is `spack install py-numpy ^cray-libsci`. To use these packages we can add them as externals in `$SPACK_ENV/packages.yaml`. **We included what is below in step 8 above.**
```yaml
packages:
py-numpy:
externals:
- spec: py-numpy@1.20.1
modules:
- cray-libsci/21.08.1.2
- cray-python/3.9.4.1
prefix: /opt/cray/pe/python/3.9.4.1/
buildable: False
py-scipy:
externals:
- spec: py-scipy@1.6.2
modules:
- cray-libsci/21.08.1.2
- cray-python/3.9.4.1
prefix: /opt/cray/pe/python/3.9.4.1/
buildable: False
```
- This is no longer necessary with an upstream fix to `py-numpy` and `py-scipy` merged into Spack.
- The same _could_ be done with `mpi4py` built against `cray-mpich`. However this doesn't appear to work at all for the Cray or AMD compiler.
```yaml
packages:
py-mpi4py:
externals:
- spec: py-mpi4py@3.0.3
modules:
- cray-mpich/8.1.9
- cray-python/3.9.4.1
prefix: /opt/cray/pe/python/3.9.4.1/
buildable: False
```
## AMD compiler (aocc) workarounds:
### Installation with Cray libsci
Compiler wrappers are missing for Cray MPICH when the AMD compiler is loaded. To work around this issue we pretend that the Cray MPICH is an older version [this bug](https://github.com/spack/spack/issues/29459), in **step 8** of the install _add_:
```yaml
packages:
cray-mpich:
externals:
# Spec line has old Cray MPICH version 8.1.6
- spec: cray-mpich@8.1.6%aocc@3.0.0
modules:
- PrgEnv-aocc
- aocc/3.0.0
- cray-mpich/8.1.9
# Modules and prefix point to version 8.1.9
prefix: /opt/cray/pe/mpich/8.1.9/ofi/aocc/3.0
```
In **step 9** the spec for installation is:
```
py-firedrake@develop \
%aocc@3.0.0 \
^python@3.9.4.1 \
^cray-mpich@8.1.6%aocc@3.0.0 \
^cray-libsci@21.04.1.1
```
Note the apparently incorrect version of MPICH for AOCC is deliberate!
### Alternative instalation with AOCL
If you wish to use AOCL in place of Cray libsci:
Manually install `py-numpy` and `py-scipy` first as they both need patching
```bash
spack install py-numpy@1.22.3%aocc@3.0.0 ^amdblis@3.1 ^amdlibflame@3.1
```
```bash
patchelf --add-needed /work/y07/shared/libs/core/aocl/3.1/AOCC/3.0/lib/libblis.so.4 /path/to/py-numpy-1.22.3-<...hash...>/lib/python3.9/site-packages/numpy/linalg/_umath_linalg.cpython-39-x86_64-linux-gnu.so
```
```bash
spack install py-scipy@1.8.0%aocc@3.0.0 ^py-numpy@1.22.3%aocc@3.0.0 ^amdblis ^amdlibflame
```
```bash
patchelf --add-needed /work/y07/shared/libs/core/aocl/3.1/AOCC/3.0/lib/libblis.so.4 /path/to/py-scipy-1.8.0-<...hash...>/lib/python3.9/site-packages/numpy/linalg/*.so
```
In **step 8** create `$SPACK_ENV/packages.yaml` with contents:
```yaml
packages:
all:
compiler: [aocc@3.0.0]
providers:
mpi: [cray-mpich]
blas: [amdblis]
lapack: [amdlibflame]
scalapack: [amdscalapack]
cray-mpich:
externals:
# Spec line has old Cray MPICH version 8.1.6
- spec: cray-mpich@8.1.6%aocc@3.0.0
modules:
- PrgEnv-aocc
- aocc/3.0.0
- cray-mpich/8.1.9
# Modules and prefix point to version 8.1.9
prefix: /opt/cray/pe/mpich/8.1.9/ofi/aocc/3.0
netlib-scalapack:
externals:
# Pretend cray-libsci is netlib-scalapack
- spec: netlib-scalapack@21.08.1.2
modules:
- cray-libsci/21.08.1.2
buildable: False
```
In **step 9** the spec for installation is:
```
py-firedrake@develop \
%aocc@3.0.0 \
^python@3.9.4.1 \
^cray-mpich@8.1.6%aocc@3.0.0 \
^py-numpy@1.22.3 \
^py-scipy@1.8.0 \
^amdblis@3.1 \
^amdlibflame@3.1 \
^amdscalapack@3.1 \
^firedrake.petsc@develop+fftw \
cflags=\"-O3 -march=native -mtune=native\" \
cxxflags=\"-O3 -march=native -mtune=native\" \
fflags=\"-O3 -march=native -mtune=native -ffree-line-length-512\"
```
Note the specification of later versions of numpy and scipy to force building these against AOCL rather than using the system versions built against Cray libsci.
### Runtime
??? Fixed by above?
In addition, because `cc` barfs a warning message PYOP2's built in compiler detection doesn't work:
```
> cc --version
Warning:
Headers and libraries from cray-libsci/21.08.1.2 will be ignored because depends on PE_MPICH which is unavailable..
AMD clang version 12.0.0 (CLANG: AOCC_3.0.0-Build#78 2020_12_10) (based on LLVM Mirror.Version.12.0.0)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/AMD/aocc-compiler-3.0.0/bin
```
To correct _this_ we must manually set
```bash
unset PYOP2_LD
export PYOP2_CC=/opt/cray/pe/craype/2.7.6/bin/cc
export PYOP2_CXX=/opt/cray/pe/craype/2.7.6/bin/CC
export PYOP2_CFLAGS="-I${CRAY_MPICH_DIR}/include/"
export PYOP2_LDFLAGS='-shared'
```
## Cray compiler (cce) workarounds:
The Cray compiler does not work and gives spurious numerical results. The workaround steps to get to this point are identical to the AMD compiler (aocc) workarounds above, but `PYOP2_LDFLAGS` need not be set.
### Installation
```
py-firedrake@develop \
%cce@12.0.3 \
^python@3.9.4.1 \
^cray-mpich@8.1.9%cce@12.0.3 \
^cray-libsci@21.04.1.1
```
### Runtime
Setting `export PYOP2_DEBUG=1` will allow tests to pass, we should work out which optimisation flags need to be turned off to get things to work!
## Interoperability with `pip`
If you pip install a Python package on ARCHER2 with or without the Spack environment active, it will get installed as a user, and as such be located in the `$HOME` directory. This is an issue since the `$HOME` directory is not mounted on the compute nodes.
The obvious solution to this is not to use pip to install the package and instead attempt to add it to the environment. If the PyPI package is named `foo`, the corresponding Spack package is likely called `py-foo`. Spack's builtin repository can be searched by executing `spack list foo` or `spack list py-foo`.
If the package is not present the correct solution is to create one and upstream it into Spack's builtin package repository. There are instructions in Spack's documentation on how to do this.
Under no circumstance's should the following hack be used:
```bash
export PIP_TARGET=$SPACK_ENV/.spack-env/view/lib/python$(python -c "import sys; print('.'.join(map(str,sys.version_info[0:2])))")/site-packages/
```
You have been warned.
## Additional notes:
### Environment
We recommend using GCC 10.2.0 since:
- The newer GCC 11.2.0 is misconfiured (see below).
- AMD (AOCC) compilers are missing wrappers in cray-mpich (although there is a workaround, see below).
- Cray compilers produce spurious numerical errors (under investigation).
---
GCC 11.2.0 fails to build superlu-dist with error
```
undefined symbol std::__throw_bad_array_new_length()@GLIBCXX_3.4.29
```
related to broken libstdc++.so, so we use 10.2.0 instead.
---
As a result of an incomplete module for xpmem, you will see:
```
> cc --version
Warning:
Headers and libraries from cray-libsci/21.08.1.2 will be ignored because depends on PE_MPICH which is unavailable..
Error invoking pkg-config!
Package cray-xpmem was not found in the pkg-config search path.
Perhaps you should add the directory containing `cray-xpmem.pc'
to the PKG_CONFIG_PATH environment variable
Package 'cray-xpmem', required by 'virtual:world', not found
```
To correct the issue:
```bash
export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/opt/cray/xpmem/2.2.40-7.0.1.0_2.7__g1d7a24d.shasta/lib64/pkgconfig
```
---
As a result of an incomplete module for cray-mpich, you will see:
```
> ftn -dumpversion
Error invoking pkg-config!
Package mpichf90 was not found in the pkg-config search path.
Perhaps you should add the directory containing `mpichf90.pc'
to the PKG_CONFIG_PATH environment variable
No package 'mpichf90' found
```
To correct the issue:
```bash
export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/opt/cray/pe/mpich/8.1.9/ofi/gnu/9.1/lib/pkgconfig
```
for GNU compilers, or:
```bash
export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/opt/cray/pe/mpich/8.1.9/ofi/aocc/3.0/lib/pkgconfig/
```
for the AMD compilers.
---
Issue:
```
> ftn -dumpversion
Warning:
Headers and libraries from cray-libsci/21.08.1.2 will be ignored because depends on PE_MPICH which is unavailable..
10.2.0
```
Fix:
```bash
export PE_PKGCONFIG_PRODUCTS=$PE_PKGCONFIG_PRODUCTS:PE_MPICH
```
However, this doesn't work for spack loading the module as there is no way to automatically set the environment variable. This really needs a fix at the system level!
---
**Note that the above issues may in fact come from Spack installing its own version of `pkgconf`, rather than using the system `pkg-config`. This requires further investigation.**
### Configuration
`$SPACK_USER_CONFIG_PATH/cray/compilers.yaml` and `$SPACK_USER_CONFIG_PATH/packages.yaml` can be initially populated with:
```bash
spack compiler find
spack external find
```
There are issues with this:
- Some of the compiler specs are wrong
- Only finds buildtools not modules or other libraries
- The `texinfo` package causes builds to fail
### Testing
Example jobscript
```bash
#!/bin/bash
#SBATCH -p standard
#SBATCH -A [budget code]
#SBATCH -J spack
#SBATCH --nodes=1
#SBATCH --cpus-per-task=1
#SBATCH --qos=standard
#SBATCH -t 0:10:00
myScript="run.sh"
module purge
module load load-epcc-module
module load PrgEnv-gnu/8.1.0
module swap gcc gcc/10.2.0
module swap cray-mpich cray-mpich/8.1.9
module swap cray-dsmml cray-dsmml/0.2.1
module swap cray-libsci cray-libsci/21.08.1.2
module load xpmem
module load cmake/3.21.3
module load cray-python/3.9.4.1
export HOME=/work/[budget code]/[budget code]/[$USER]
source /work/[budget code]/[budget code]/[$USER]/workspace/spack/share/spack/setup-env.sh
spack env activate /work/[budget code]/[budget code]/[$USER]/workspace/firedrake
./${myScript}
```
# June 2023 Update:
```
module purge
module load load-epcc-module
# Load programming environment and compiler
module load PrgEnv-gnu/8.3.3
# module swap gcc gcc/10.2.0 -> 11.2.0 ???
# module swap cray-mpich cray-mpich/8.1.23
# module swap cray-dsmml cray-dsmml/0.2.2
# module swap cray-libsci cray-libsci/22.12.1.1
module load xpmem
module load cmake/3.21.3
module load cray-python/3.9.13.1
```
`$SPACK_USER_CONFIG/linux/compilers.yaml`:
```
compilers:
- compiler:
spec: gcc@=10.2.0
paths:
cc: cc
cxx: CC
f77: ftn
fc: ftn
flags: {}
operating_system: sles15
target: any
modules:
- PrgEnv-gnu
- gcc/10.2.0
environment: {}
extra_rpaths: []
- compiler:
spec: gcc@=11.2.0
paths:
cc: cc
cxx: CC
f77: ftn
fc: ftn
flags: {}
operating_system: sles15
target: any
modules:
- PrgEnv-gnu
- gcc/11.2.0
environment: {}
extra_rpaths: []
```
`$SPACK_USER_CONFIG/packages.yaml`:
```
packages:
all:
providers:
mpi: [cray-mpich]
cray-mpich:
externals:
# Cray-mpich for GCC 10
- spec: cray-mpich@8.1.23%gcc@10.2.0
modules:
- PrgEnv-gnu
- gcc/10.2.0
- craype-network-ofi
- cray-mpich/8.1.23
prefix: /opt/cray/pe/mpich/8.1.23/ofi/gnu/9.1
- spec: cray-mpich@88.1.9%gcc@10.2.0
modules:
- PrgEnv-gnu
- gcc/10.2.0
- craype-network-ucx
- cray-mpich/8.1.23
prefix: /opt/cray/pe/mpich/8.1.23/ucx/gnu/9.1
# Cray-mpich for GCC 11
- spec: cray-mpich@8.1.23%gcc@11.2.0
modules:
- PrgEnv-gnu
- gcc/11.2.0
- craype-network-ofi
- cray-mpich/8.1.23
prefix: /opt/cray/pe/mpich/8.1.23/ofi/gnu/9.1
- spec: cray-mpich@88.1.9%gcc@11.2.0
modules:
- PrgEnv-gnu
- gcc/11.2.0
- craype-network-ucx
- cray-mpich/8.1.23
prefix: /opt/cray/pe/mpich/8.1.23/ucx/gnu/9.1
cray-libsci:
externals:
- spec: cray-libsci@22.12.1.1
modules:
- cray-libsci/22.12.1.1
# Hopefully no longer necessary...
# environment:
# append_path:
# - PE_PKGCONFIG_PRODUCTS: PE_MPICH
buildable: False
python:
externals:
- spec: python@3.9.13.1
modules:
- cray-python/3.9.13.1
buildable: False
# The following were detected by `spack external find`
groff:
externals:
- spec: groff@1.22.4
prefix: /usr
gawk:
externals:
- spec: gawk@4.2.1
prefix: /usr
subversion:
externals:
- spec: subversion@1.14.1
prefix: /usr
bison:
externals:
- spec: bison@3.0.4
prefix: /usr
autoconf:
externals:
- spec: autoconf@2.69
prefix: /usr
diffutils:
externals:
- spec: diffutils@3.6
prefix: /usr
tar:
externals:
- spec: tar@1.34
prefix: /usr
flex:
externals:
- spec: flex@2.6.4+lex
prefix: /usr
curl:
externals:
- spec: curl@7.79.1+gssapi+ldap+nghttp2
prefix: /usr
automake:
externals:
- spec: automake@1.15.1
prefix: /usr
# System texinfo causes issues!
# texinfo:
# externals:
# - spec: texinfo@6.5
# prefix: /usr
gmake:
externals:
- spec: gmake@4.2.1
prefix: /usr
cmake:
externals:
- spec: cmake@3.20.4
prefix: /usr
- spec: cmake@3.21.3
prefix: /work/y07/shared/utils/core/cmake/3.21.3
findutils:
externals:
- spec: findutils@4.8.0
prefix: /usr
# We MUST use Cray's pkg-config not build our own
pkg-config:
externals:
- spec: pkg-config@0.29.2
prefix: /usr
# No building!
buildable: False
# This applies to pkgconf too!
pkgconf:
externals:
- spec: pkg-config@0.29.2
prefix: /usr
buildable: False
openssl:
externals:
- spec: openssl@1.1.1l
prefix: /usr
coreutils:
externals:
- spec: coreutils@8.32
prefix: /usr
binutils:
externals:
- spec: binutils@2.39.0
prefix: /usr
openssh:
externals:
- spec: openssh@8.4p1
prefix: /usr
m4:
externals:
- spec: m4@1.4.18
prefix: /usr
libtool:
externals:
- spec: libtool@2.4.6
prefix: /usr
git:
externals:
- spec: git@2.35.3~tcltk
prefix: /usr
```