# Spack install Firedrake on ARCHER2 ###### tags: `spack` `archer2` We will work with the in a directory `/work/[budget code]/[budget code]/[$USER]/workspace`, where `[budget code]` is replaced by your project code and `[$USER]` is your username. ## Prerequisites You should familiarise yourself with as much of Spack's functionality as possible. To achieve the same flexibility as we have on desktop installs the Firedrake Spack install leverages a lot of Spack's features. More information about Spack can be found on their [website](https://spack.readthedocs.io/en/latest/index.html). ## Spack Start in a directory on the WORK filesystem, not HOME as it is not accessible from the compute nodes. You may wish to set `SPACK_USER_CONFIG_PATH` which defaults to `~/.spack` to change where spack looks for user level configuration files. Changing `SPACK_USER_CONFIG_PATH` isn't necessary if everything is correctly configured within the spack environment. ### Environment We recommend using the GNU toolchain and GCC compilers at version 10.2.0, see [additional notes](#Additional-notes) for more info. ```bash git clone -c feature.manyFiles=true git@github.com:spack/spack.git module purge module load load-epcc-module # Load programming environment and compiler module load PrgEnv-gnu/8.1.0 module swap gcc gcc/10.2.0 module swap cray-mpich cray-mpich/8.1.9 module swap cray-dsmml cray-dsmml/0.2.1 module swap cray-libsci cray-libsci/21.08.1.2 module load xpmem module load cmake/3.21.3 module load cray-python/3.9.4.1 . spack/share/spack/setup-env.sh ``` If you want to use a different BLAS and LAPACK and Scalapack library you don't need to load cray-libsci and you should run `module unload cray-libsci`. For instance the AMD optimised CPU libraries (AOCL) could be loaded instead `module load aocl/3.1` If you _really_ want to try the other compilers replace the two isolated lines under `# Load programming environment and compiler` with ``` module load PrgEnv-aocc/8.1.0 module swap aocc aocc/3.0.0 # or module load PrgEnv-cray/8.1.0 module swap cce cce/12.0.3 ``` for the AMD or Cray compilers respectively. ### Configure spack 1. Populate `$SPACK_USER_CONFIG_PATH/cray/compilers.yaml` with the following YAML. Note that this contains most of the available compilers, you only need to configure for the compilers you will use. <details> <summary> <code>$SPACK_USER_CONFIG_PATH/cray/compilers.yaml</code> </summary> ```yaml compilers: - compiler: spec: aocc@2.2.0 paths: cc: cc cxx: CC f77: ftn fc: ftn flags: cflags: null cxxflags: null fflags: null operating_system: sles15 target: any modules: - PrgEnv-aocc - aocc/2.2.0 environment: {} extra_rpaths: [] - compiler: spec: aocc@2.2.0.1 paths: cc: cc cxx: CC f77: ftn fc: ftn flags: cflags: null cxxflags: null fflags: null operating_system: sles15 target: any modules: - PrgEnv-aocc - aocc/2.2.0.1 environment: {} extra_rpaths: [] - compiler: spec: aocc@3.0.0 paths: cc: cc cxx: CC f77: ftn fc: ftn flags: cflags: -Wno-unused-command-line-argument -mllvm -eliminate-similar-expr=false cxxflags: -Wno-unused-command-line-argument -mllvm -eliminate-similar-expr=false fflags: -Wno-unused-command-line-argument -mllvm -eliminate-similar-expr=false ldflags: -Wl,-rpath,/opt/cray/libfabric/1.11.0.4.71/lib64 -L/opt/cray/libfabric/1.11.0.4.71/lib64 -lfabric operating_system: sles15 target: any modules: - PrgEnv-aocc - aocc/3.0.0 environment: {} extra_rpaths: [] - compiler: spec: cce@11.0.4 paths: cc: cc cxx: CC f77: ftn fc: ftn flags: {} operating_system: sles15 target: any modules: - PrgEnv-cray - cce/11.0.4 environment: {} extra_rpaths: [] - compiler: spec: cce@12.0.3 paths: cc: cc cxx: CC f77: ftn fc: ftn flags: {} operating_system: sles15 target: any modules: - PrgEnv-cray - cce/12.0.3 environment: {} extra_rpaths: [] - compiler: spec: gcc@9.3.0 paths: cc: cc cxx: CC f77: ftn fc: ftn flags: {} operating_system: sles15 target: any modules: - PrgEnv-gnu - gcc/9.3.0 environment: {} extra_rpaths: [] - compiler: spec: gcc@10.2.0 paths: cc: cc cxx: CC f77: ftn fc: ftn flags: {} operating_system: sles15 target: any modules: - PrgEnv-gnu - gcc/10.2.0 environment: {} extra_rpaths: [] - compiler: spec: gcc@10.3.0 paths: cc: cc cxx: CC f77: ftn fc: ftn flags: {} operating_system: sles15 target: any modules: - PrgEnv-gnu - gcc/10.3.0 environment: {} extra_rpaths: [] - compiler: spec: gcc@11.2.0 paths: cc: cc cxx: CC f77: ftn fc: ftn flags: {} operating_system: sles15 target: any modules: - PrgEnv-gnu - gcc/11.2.0 environment: {} extra_rpaths: [] - compiler: spec: rocmcc@2.14-nosmp paths: cc: cc cxx: CC f77: ftn fc: ftn flags: {} operating_system: sles15 target: any modules: - PrgEnv-amd - rocmcc/2.14-nosmp environment: {} extra_rpaths: [] - compiler: spec: rocmcc@2.14 paths: cc: cc cxx: CC f77: ftn fc: ftn flags: {} operating_system: sles15 target: any modules: - PrgEnv-amd - rocmcc/2.14 environment: {} extra_rpaths: [] ``` </details> 2. Populate `$SPACK_USER_CONFIG_PATH/packages.yaml` with the following <details> <summary> <code>$SPACK_USER_CONFIG_PATH/packages.yaml</code> </summary> ```yaml packages: all: providers: mpi: [cray-mpich] cray-mpich: externals: - spec: cray-mpich@8.1.9%gcc@10.2.0 modules: - PrgEnv-gnu - gcc/10.2.0 - cray-mpich/8.1.9 - spec: cray-mpich@8.1.6%aocc@3.0.0 # Pretend mpich is version 8.1.6 due to the following bug in spack # https://github.com/spack/spack/issues/29459 # Only with aocc compiler modules: - PrgEnv-aocc - aocc/3.0.0 - cray-mpich/8.1.9 prefix: /opt/cray/pe/mpich/8.1.9/ofi/aocc/3.0 - spec: cray-mpich@8.1.9%cce@12.0.3 modules: - PrgEnv-cray - cce/12.0.3 - cray-mpich/8.1.9 buildable: False cray-libsci: externals: - spec: cray-libsci@21.04.1.1 modules: - cray-libsci/21.04.1.1 - spec: cray-libsci@21.08.1.2 modules: - cray-libsci/21.08.1.2 environment: append_path: - PE_PKGCONFIG_PRODUCTS: PE_MPICH buildable: False amdblis: externals: - spec: amdblis@3.1 modules: - aocl/3.1 buildable: False amdfftw: externals: - spec: amdfftw@3.1 modules: - aocl/3.1 buildable: False amdlibflame: externals: - spec: amdlibflame@3.1 modules: - aocl/3.1 buildable: False amdlibm: externals: - spec: amdlibm@3.1 modules: - aocl/3.1 buildable: False amdscalapack: externals: - spec: amdscalapack@3.1 modules: - aocl/3.1 buildable: False aocl-sparse: externals: - spec: aocl-sparse@3.1 modules: - aocl/3.1 buildable: False python: externals: - spec: python@3.9.4.1 modules: - cray-python/3.9.4.1 - spec: python@3.8.5.0 modules: - cray-python/3.8.5.0 buildable: False flex: externals: - spec: flex@2.6.4+lex prefix: /usr # System texinfo causes issues! # texinfo: # externals: # - spec: texinfo@6.5 # prefix: /usr automake: externals: - spec: automake@1.15.1 prefix: /usr binutils: externals: - spec: binutils@2.35.1 prefix: /usr bison: externals: - spec: bison@3.0.4 prefix: /usr libtool: externals: - spec: libtool@2.4.6 prefix: /usr m4: externals: - spec: m4@1.4.18 prefix: /usr openssh: externals: - spec: openssh@7.9p1 prefix: /usr gawk: externals: - spec: gawk@4.2.1 prefix: /usr pkg-config: externals: - spec: pkg-config@0.29.2 prefix: /usr buildable: False pkgconf: externals: - spec: pkg-config@0.29.2 prefix: /usr buildable: False openssl: externals: - spec: openssl@1.1.0i-fips prefix: /usr groff: externals: - spec: groff@1.22.3 prefix: /usr autoconf: externals: - spec: autoconf@2.69 prefix: /usr findutils: externals: - spec: findutils@4.6.0 prefix: /usr git: externals: - spec: git@2.26.2~tcltk prefix: /usr tar: externals: - spec: tar@1.30 prefix: /usr subversion: externals: - spec: subversion@1.10.0 prefix: /usr cmake: externals: - spec: cmake@3.10.2 prefix: /usr - spec: cmake@3.21.3 prefix: /work/y07/shared/utils/core/cmake/3.21.3 gmake: externals: - spec: gmake@4.2.1 prefix: /usr diffutils: externals: - spec: diffutils@3.6 prefix: /usr ``` </details> ## Firedrake ### Installation 1. Clone the repo: ```bash git clone https://github.com/firedrakeproject/firedrake-spack.git # or git clone git@github.com:firedrakeproject/firedrake-spack.git ``` 1. Add the repository to spack `spack repo add <repo directory> ` 1. Create an spack environment `spack env create -d ./firedrake` 1. Activate that environment `spack env activate -p ./firedrake` 1. To avoid a bunch of errors add a whole bunch of packages to the development package list: ``` spack develop py-firedrake@develop spack develop libsupermesh@develop spack develop petsc@develop spack develop chaco@petsc spack develop py-fiat@develop spack develop py-finat@develop spack develop py-islpy@develop spack develop py-petsc4py@develop spack develop py-pyadjoint@develop spack develop py-pyop2@develop spack develop py-coffee@develop spack develop py-loopy@develop spack develop py-cgen@develop spack develop py-codepy@develop spack develop py-genpy@develop spack develop py-tsfc@develop spack develop py-ufl@develop ``` 1. To work around the fact that `/home` is not accessible from compute nodes the location of the firedrake repo needs to be set inside `$SPACK_ENV/spack.yaml`: ```yaml spack: repos: - /work/[budget code]/[budget code]/[$USER]/workspace/firedrake-spack ``` Where the path should point to the repo cloned in step 1. 1. Then custom configure the packages inside `$SPACK_ENV/spack.yaml`: ```yaml spack: include: - packages.yaml ``` 1. And create `$SPACK_ENV/packages.yaml` with contents: ```yaml packages: all: compiler: [gcc@10.2.0] providers: mpi: [cray-mpich] blas: [cray-libsci] lapack: [cray-libsci] scalapack: [cray-libsci] netlib-scalapack: externals: # Pretend cray-libsci is netlib-scalapack - spec: netlib-scalapack@21.08.1.2 modules: - cray-libsci/21.08.1.2 buildable: False py-numpy: externals: - spec: py-numpy@1.20.1 modules: - cray-libsci/21.08.1.2 - cray-python/3.9.4.1 prefix: /opt/cray/pe/python/3.9.4.1/ buildable: False py-scipy: externals: - spec: py-scipy@1.6.2 modules: - cray-libsci/21.08.1.2 - cray-python/3.9.4.1 prefix: /opt/cray/pe/python/3.9.4.1/ buildable: False ``` If using AMD or Cray compilers replace the compiler above (`[gcc@10.2.0]`) with `[aocc@3.0.0]` or `[cce@12.0.3]`. 1. Add Firedrake to the spec for the environment ``` spack add py-firedrake@develop \ %gcc@10.2.0 \ ^python@3.9.4.1 \ ^cray-mpich@8.1.9%gcc@10.2.0 \ ^cray-libsci@21.08.1.2 \ ^firedrake.petsc@develop+fftw \ cflags=\"-O3 -march=native -mtune=native\" \ cxxflags=\"-O3 -march=native -mtune=native\" \ fflags=\"-O3 -march=native -mtune=native -ffree-line-length-512\" ``` 1. Concretize (and log) `spack concretize -f 2>&1 | tee $SPACK_ENV/conc.log` * Splitting the concretize and install steps into 2 distinct parts allows you to review what is being installed before the lengthy installation process begins. 1. Install (and log) `spack install --fail-fast 2>&1 | tee $SPACK_ENV/spack-firedrake-install.log` ### Testing Testing must be run on a compute node. An interactive session can be started using ```bash srun --nodes=1 --exclusive --time=00:20:00 \ --partition=standard --qos=short --reservation=shortqos \ --account=[budget code] --pty /bin/bash ``` Alternatively submit a jobscript, see [additional notes](#Additional-notes). 1. Test you can import Firedrake by running `python -c "from firedrake import *"` - If this fails, before trying anything else, deactivate the environment with `spack env deactivate` and reactivate with `spack env activate -p ./firedrake` (as above) and try running `python -c "from firedrake import *"` again. This appears to be a shortcoming of spack. 1. Run the basic functionality tests: ```bash cd $SPACK_ENV/py-firedrake pytest tests/regression/ -m "not parallel" -k "poisson_strong or stokes_mini or dg_advection" ``` - When running the environment on the compute nodes the variable `$HOME` needs to be set to something like `/work/_projectcode_/_projectcode_/_username_` or `/tmp` as many dependencies expect to find configuration in `$HOME`. - We don't run parallel tests due to an issue with the current parallel pytest hook, this is in the process of being fixed. 1. Run the full test suite: ```bash cd $SPACK_ENV/src/firedrake pytest tests -m "not parallel" ``` ### General issues encountered This section contains development notes, and is only included for future development and general interest. Skip this section to read about configuring for AMD or Cray compilers. It's almost certainly better to do everything in this section on a per env basis. If this is done globally in `~/.spack/packages.yaml` it will not be possible to install other versions of these packages. - MUMPS insists on building with `netlib-scalapack`. To use `cray-libsci` we can create an external packages called `netlib-scalapack` that loads the `cray-libsci` modules and add it to `$SPACK_ENV/packages.yaml`. **This may not be a good idea! We included what is below in step 8 above as it seems to work YMMV.** ```yaml packages: netlib-scalapack: externals: # Pretend cray-libsci is netlib-scalapack - spec: netlib-scalapack@21.08.1.2 modules: - cray-libsci/21.08.1.2 buildable: False ``` - It's not really possible to build `numpy` and `scipy` using the `cray-libsci` library, that is `spack install py-numpy ^cray-libsci`. To use these packages we can add them as externals in `$SPACK_ENV/packages.yaml`. **We included what is below in step 8 above.** ```yaml packages: py-numpy: externals: - spec: py-numpy@1.20.1 modules: - cray-libsci/21.08.1.2 - cray-python/3.9.4.1 prefix: /opt/cray/pe/python/3.9.4.1/ buildable: False py-scipy: externals: - spec: py-scipy@1.6.2 modules: - cray-libsci/21.08.1.2 - cray-python/3.9.4.1 prefix: /opt/cray/pe/python/3.9.4.1/ buildable: False ``` - This is no longer necessary with an upstream fix to `py-numpy` and `py-scipy` merged into Spack. - The same _could_ be done with `mpi4py` built against `cray-mpich`. However this doesn't appear to work at all for the Cray or AMD compiler. ```yaml packages: py-mpi4py: externals: - spec: py-mpi4py@3.0.3 modules: - cray-mpich/8.1.9 - cray-python/3.9.4.1 prefix: /opt/cray/pe/python/3.9.4.1/ buildable: False ``` ## AMD compiler (aocc) workarounds: ### Installation with Cray libsci Compiler wrappers are missing for Cray MPICH when the AMD compiler is loaded. To work around this issue we pretend that the Cray MPICH is an older version [this bug](https://github.com/spack/spack/issues/29459), in **step 8** of the install _add_: ```yaml packages: cray-mpich: externals: # Spec line has old Cray MPICH version 8.1.6 - spec: cray-mpich@8.1.6%aocc@3.0.0 modules: - PrgEnv-aocc - aocc/3.0.0 - cray-mpich/8.1.9 # Modules and prefix point to version 8.1.9 prefix: /opt/cray/pe/mpich/8.1.9/ofi/aocc/3.0 ``` In **step 9** the spec for installation is: ``` py-firedrake@develop \ %aocc@3.0.0 \ ^python@3.9.4.1 \ ^cray-mpich@8.1.6%aocc@3.0.0 \ ^cray-libsci@21.04.1.1 ``` Note the apparently incorrect version of MPICH for AOCC is deliberate! ### Alternative instalation with AOCL If you wish to use AOCL in place of Cray libsci: Manually install `py-numpy` and `py-scipy` first as they both need patching ```bash spack install py-numpy@1.22.3%aocc@3.0.0 ^amdblis@3.1 ^amdlibflame@3.1 ``` ```bash patchelf --add-needed /work/y07/shared/libs/core/aocl/3.1/AOCC/3.0/lib/libblis.so.4 /path/to/py-numpy-1.22.3-<...hash...>/lib/python3.9/site-packages/numpy/linalg/_umath_linalg.cpython-39-x86_64-linux-gnu.so ``` ```bash spack install py-scipy@1.8.0%aocc@3.0.0 ^py-numpy@1.22.3%aocc@3.0.0 ^amdblis ^amdlibflame ``` ```bash patchelf --add-needed /work/y07/shared/libs/core/aocl/3.1/AOCC/3.0/lib/libblis.so.4 /path/to/py-scipy-1.8.0-<...hash...>/lib/python3.9/site-packages/numpy/linalg/*.so ``` In **step 8** create `$SPACK_ENV/packages.yaml` with contents: ```yaml packages: all: compiler: [aocc@3.0.0] providers: mpi: [cray-mpich] blas: [amdblis] lapack: [amdlibflame] scalapack: [amdscalapack] cray-mpich: externals: # Spec line has old Cray MPICH version 8.1.6 - spec: cray-mpich@8.1.6%aocc@3.0.0 modules: - PrgEnv-aocc - aocc/3.0.0 - cray-mpich/8.1.9 # Modules and prefix point to version 8.1.9 prefix: /opt/cray/pe/mpich/8.1.9/ofi/aocc/3.0 netlib-scalapack: externals: # Pretend cray-libsci is netlib-scalapack - spec: netlib-scalapack@21.08.1.2 modules: - cray-libsci/21.08.1.2 buildable: False ``` In **step 9** the spec for installation is: ``` py-firedrake@develop \ %aocc@3.0.0 \ ^python@3.9.4.1 \ ^cray-mpich@8.1.6%aocc@3.0.0 \ ^py-numpy@1.22.3 \ ^py-scipy@1.8.0 \ ^amdblis@3.1 \ ^amdlibflame@3.1 \ ^amdscalapack@3.1 \ ^firedrake.petsc@develop+fftw \ cflags=\"-O3 -march=native -mtune=native\" \ cxxflags=\"-O3 -march=native -mtune=native\" \ fflags=\"-O3 -march=native -mtune=native -ffree-line-length-512\" ``` Note the specification of later versions of numpy and scipy to force building these against AOCL rather than using the system versions built against Cray libsci. ### Runtime ??? Fixed by above? In addition, because `cc` barfs a warning message PYOP2's built in compiler detection doesn't work: ``` > cc --version Warning: Headers and libraries from cray-libsci/21.08.1.2 will be ignored because depends on PE_MPICH which is unavailable.. AMD clang version 12.0.0 (CLANG: AOCC_3.0.0-Build#78 2020_12_10) (based on LLVM Mirror.Version.12.0.0) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /opt/AMD/aocc-compiler-3.0.0/bin ``` To correct _this_ we must manually set ```bash unset PYOP2_LD export PYOP2_CC=/opt/cray/pe/craype/2.7.6/bin/cc export PYOP2_CXX=/opt/cray/pe/craype/2.7.6/bin/CC export PYOP2_CFLAGS="-I${CRAY_MPICH_DIR}/include/" export PYOP2_LDFLAGS='-shared' ``` ## Cray compiler (cce) workarounds: The Cray compiler does not work and gives spurious numerical results. The workaround steps to get to this point are identical to the AMD compiler (aocc) workarounds above, but `PYOP2_LDFLAGS` need not be set. ### Installation ``` py-firedrake@develop \ %cce@12.0.3 \ ^python@3.9.4.1 \ ^cray-mpich@8.1.9%cce@12.0.3 \ ^cray-libsci@21.04.1.1 ``` ### Runtime Setting `export PYOP2_DEBUG=1` will allow tests to pass, we should work out which optimisation flags need to be turned off to get things to work! ## Interoperability with `pip` If you pip install a Python package on ARCHER2 with or without the Spack environment active, it will get installed as a user, and as such be located in the `$HOME` directory. This is an issue since the `$HOME` directory is not mounted on the compute nodes. The obvious solution to this is not to use pip to install the package and instead attempt to add it to the environment. If the PyPI package is named `foo`, the corresponding Spack package is likely called `py-foo`. Spack's builtin repository can be searched by executing `spack list foo` or `spack list py-foo`. If the package is not present the correct solution is to create one and upstream it into Spack's builtin package repository. There are instructions in Spack's documentation on how to do this. Under no circumstance's should the following hack be used: ```bash export PIP_TARGET=$SPACK_ENV/.spack-env/view/lib/python$(python -c "import sys; print('.'.join(map(str,sys.version_info[0:2])))")/site-packages/ ``` You have been warned. ## Additional notes: ### Environment We recommend using GCC 10.2.0 since: - The newer GCC 11.2.0 is misconfiured (see below). - AMD (AOCC) compilers are missing wrappers in cray-mpich (although there is a workaround, see below). - Cray compilers produce spurious numerical errors (under investigation). --- GCC 11.2.0 fails to build superlu-dist with error ``` undefined symbol std::__throw_bad_array_new_length()@GLIBCXX_3.4.29 ``` related to broken libstdc++.so, so we use 10.2.0 instead. --- As a result of an incomplete module for xpmem, you will see: ``` > cc --version Warning: Headers and libraries from cray-libsci/21.08.1.2 will be ignored because depends on PE_MPICH which is unavailable.. Error invoking pkg-config! Package cray-xpmem was not found in the pkg-config search path. Perhaps you should add the directory containing `cray-xpmem.pc' to the PKG_CONFIG_PATH environment variable Package 'cray-xpmem', required by 'virtual:world', not found ``` To correct the issue: ```bash export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/opt/cray/xpmem/2.2.40-7.0.1.0_2.7__g1d7a24d.shasta/lib64/pkgconfig ``` --- As a result of an incomplete module for cray-mpich, you will see: ``` > ftn -dumpversion Error invoking pkg-config! Package mpichf90 was not found in the pkg-config search path. Perhaps you should add the directory containing `mpichf90.pc' to the PKG_CONFIG_PATH environment variable No package 'mpichf90' found ``` To correct the issue: ```bash export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/opt/cray/pe/mpich/8.1.9/ofi/gnu/9.1/lib/pkgconfig ``` for GNU compilers, or: ```bash export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/opt/cray/pe/mpich/8.1.9/ofi/aocc/3.0/lib/pkgconfig/ ``` for the AMD compilers. --- Issue: ``` > ftn -dumpversion Warning: Headers and libraries from cray-libsci/21.08.1.2 will be ignored because depends on PE_MPICH which is unavailable.. 10.2.0 ``` Fix: ```bash export PE_PKGCONFIG_PRODUCTS=$PE_PKGCONFIG_PRODUCTS:PE_MPICH ``` However, this doesn't work for spack loading the module as there is no way to automatically set the environment variable. This really needs a fix at the system level! --- **Note that the above issues may in fact come from Spack installing its own version of `pkgconf`, rather than using the system `pkg-config`. This requires further investigation.** ### Configuration `$SPACK_USER_CONFIG_PATH/cray/compilers.yaml` and `$SPACK_USER_CONFIG_PATH/packages.yaml` can be initially populated with: ```bash spack compiler find spack external find ``` There are issues with this: - Some of the compiler specs are wrong - Only finds buildtools not modules or other libraries - The `texinfo` package causes builds to fail ### Testing Example jobscript ```bash #!/bin/bash #SBATCH -p standard #SBATCH -A [budget code] #SBATCH -J spack #SBATCH --nodes=1 #SBATCH --cpus-per-task=1 #SBATCH --qos=standard #SBATCH -t 0:10:00 myScript="run.sh" module purge module load load-epcc-module module load PrgEnv-gnu/8.1.0 module swap gcc gcc/10.2.0 module swap cray-mpich cray-mpich/8.1.9 module swap cray-dsmml cray-dsmml/0.2.1 module swap cray-libsci cray-libsci/21.08.1.2 module load xpmem module load cmake/3.21.3 module load cray-python/3.9.4.1 export HOME=/work/[budget code]/[budget code]/[$USER] source /work/[budget code]/[budget code]/[$USER]/workspace/spack/share/spack/setup-env.sh spack env activate /work/[budget code]/[budget code]/[$USER]/workspace/firedrake ./${myScript} ``` # June 2023 Update: ``` module purge module load load-epcc-module # Load programming environment and compiler module load PrgEnv-gnu/8.3.3 # module swap gcc gcc/10.2.0 -> 11.2.0 ??? # module swap cray-mpich cray-mpich/8.1.23 # module swap cray-dsmml cray-dsmml/0.2.2 # module swap cray-libsci cray-libsci/22.12.1.1 module load xpmem module load cmake/3.21.3 module load cray-python/3.9.13.1 ``` `$SPACK_USER_CONFIG/linux/compilers.yaml`: ``` compilers: - compiler: spec: gcc@=10.2.0 paths: cc: cc cxx: CC f77: ftn fc: ftn flags: {} operating_system: sles15 target: any modules: - PrgEnv-gnu - gcc/10.2.0 environment: {} extra_rpaths: [] - compiler: spec: gcc@=11.2.0 paths: cc: cc cxx: CC f77: ftn fc: ftn flags: {} operating_system: sles15 target: any modules: - PrgEnv-gnu - gcc/11.2.0 environment: {} extra_rpaths: [] ``` `$SPACK_USER_CONFIG/packages.yaml`: ``` packages: all: providers: mpi: [cray-mpich] cray-mpich: externals: # Cray-mpich for GCC 10 - spec: cray-mpich@8.1.23%gcc@10.2.0 modules: - PrgEnv-gnu - gcc/10.2.0 - craype-network-ofi - cray-mpich/8.1.23 prefix: /opt/cray/pe/mpich/8.1.23/ofi/gnu/9.1 - spec: cray-mpich@88.1.9%gcc@10.2.0 modules: - PrgEnv-gnu - gcc/10.2.0 - craype-network-ucx - cray-mpich/8.1.23 prefix: /opt/cray/pe/mpich/8.1.23/ucx/gnu/9.1 # Cray-mpich for GCC 11 - spec: cray-mpich@8.1.23%gcc@11.2.0 modules: - PrgEnv-gnu - gcc/11.2.0 - craype-network-ofi - cray-mpich/8.1.23 prefix: /opt/cray/pe/mpich/8.1.23/ofi/gnu/9.1 - spec: cray-mpich@88.1.9%gcc@11.2.0 modules: - PrgEnv-gnu - gcc/11.2.0 - craype-network-ucx - cray-mpich/8.1.23 prefix: /opt/cray/pe/mpich/8.1.23/ucx/gnu/9.1 cray-libsci: externals: - spec: cray-libsci@22.12.1.1 modules: - cray-libsci/22.12.1.1 # Hopefully no longer necessary... # environment: # append_path: # - PE_PKGCONFIG_PRODUCTS: PE_MPICH buildable: False python: externals: - spec: python@3.9.13.1 modules: - cray-python/3.9.13.1 buildable: False # The following were detected by `spack external find` groff: externals: - spec: groff@1.22.4 prefix: /usr gawk: externals: - spec: gawk@4.2.1 prefix: /usr subversion: externals: - spec: subversion@1.14.1 prefix: /usr bison: externals: - spec: bison@3.0.4 prefix: /usr autoconf: externals: - spec: autoconf@2.69 prefix: /usr diffutils: externals: - spec: diffutils@3.6 prefix: /usr tar: externals: - spec: tar@1.34 prefix: /usr flex: externals: - spec: flex@2.6.4+lex prefix: /usr curl: externals: - spec: curl@7.79.1+gssapi+ldap+nghttp2 prefix: /usr automake: externals: - spec: automake@1.15.1 prefix: /usr # System texinfo causes issues! # texinfo: # externals: # - spec: texinfo@6.5 # prefix: /usr gmake: externals: - spec: gmake@4.2.1 prefix: /usr cmake: externals: - spec: cmake@3.20.4 prefix: /usr - spec: cmake@3.21.3 prefix: /work/y07/shared/utils/core/cmake/3.21.3 findutils: externals: - spec: findutils@4.8.0 prefix: /usr # We MUST use Cray's pkg-config not build our own pkg-config: externals: - spec: pkg-config@0.29.2 prefix: /usr # No building! buildable: False # This applies to pkgconf too! pkgconf: externals: - spec: pkg-config@0.29.2 prefix: /usr buildable: False openssl: externals: - spec: openssl@1.1.1l prefix: /usr coreutils: externals: - spec: coreutils@8.32 prefix: /usr binutils: externals: - spec: binutils@2.39.0 prefix: /usr openssh: externals: - spec: openssh@8.4p1 prefix: /usr m4: externals: - spec: m4@1.4.18 prefix: /usr libtool: externals: - spec: libtool@2.4.6 prefix: /usr git: externals: - spec: git@2.35.3~tcltk prefix: /usr ```