Spack install Firedrake on Isambard

tags: spack

We have only tried installing on the XCI (ARM/ThunderX2/TX2) system, not the MACS, A64FX or Phase3 portions of the machine.

Prerequisites

You should familiarise yourself with as much of Spack's functionality as possible. To achieve the same flexibility as we have on desktop installs the Firedrake Spack install leverages a lot of Spack's features. More information about Spack can be found on their website.

Spack

Environment

  • The following clones the Spack repo, loads modules to setup the environment and activates Spack.
git clone -c feature.manyFiles=true git@github.com:spack/spack.git

module purge
module load Base-opts
module load PrgEnv-gnu
module load pmi-lib
module load cray-python/3.8.5.1
module load cray-mpich

. spack/share/spack/setup-env.sh

Configuration

  1. Populate $SPACK_USER_CONFIG_PATH/cray/compilers.yaml with
$SPACK_USER_CONFIG_PATH/cray/compilers.yaml 
compilers:
- compiler:
    spec: cce@11.0.4
    paths:
      cc: cc
      cxx: CC
      f77: ftn
      fc: ftn
    flags: {}
    operating_system: cnl7
    target: any
    modules:
    - PrgEnv-cray
    - cce/11.0.4
    environment: {}
    extra_rpaths: []
- compiler:
    spec: gcc@9.3.0
    paths:
      cc: cc
      cxx: CC
      f77: ftn
      fc: ftn
    flags: {}
    operating_system: cnl7
    target: any
    modules:
    - PrgEnv-gnu
    - gcc/9.3.0
    environment: {}
    extra_rpaths: []
  1. Populate $SPACK_USER_CONFIG_PATH/packages.yaml with
$SPACK_USER_CONFIG_PATH/packages.yaml 
packages:
  all:
    providers:
      mpi: [cray-mpich]
  pkg-config:
    externals:
    - spec: pkg-config@0.29.2
      prefix: /usr
    buildable: False
  pkgconf:
    externals:
    - spec: pkgconf@0.29.2
      prefix: /usr
    buildable: False
  cray-mpich:
    externals:
    - spec: cray-mpich@7.7.17%gcc@9.3.0
      modules:
      - PrgEnv-gnu
      - gcc/9.3.0
      - cray-mpich/7.7.17
    - spec: cray-mpich@7.7.17%cce@11.0.4
      modules:
      - PrgEnv-cray
      - cce/11.0.4
      - cray-mpich/7.7.17
    buildable: False
  cray-libsci:
    externals:
      - spec: cray-libsci@20.09.1
        modules:
        - cray-libsci/20.09.1
      - spec: cray-libsci@19.06.1
        modules:
        - cray-libsci/19.06.1
    buildable: False
  python:
    externals:
      - spec: python@3.8.5.1
        modules:
        - cray-python/3.8.5.1
      - spec: cray-python@3.7.3.2
        modules:
        - cray-python/3.7.3.2
    buildable: False
  1. Esure $SPACK_USER_CONFIG_PATH/upstreams.yaml does not exist. Upstream (system level installed Spack packages) for Isambard cause the Cray tools (compiler wrappers) to break.

Firedrake

  1. Clone the repo:

    ​​​​git clone https://github.com/firedrakeproject/firedrake-spack.git
​​​​# or
​​​​git clone git@github.com:firedrakeproject/firedrake-spack.git

  2. Add the repository to spack spack repo add <repo directory>

  3. Create an spack environment spack env create -d ./firedrake

  4. Activate that environment spack env activate -p ./firedrake

  5. To avoid a bunch of errors add a whole bunch of packages to the development package list:

    ​​​​spack develop py-firedrake@develop
​​​​spack develop libsupermesh@develop
​​​​spack develop petsc@develop
​​​​spack develop chaco@petsc
​​​​spack develop py-fiat@develop
​​​​spack develop py-finat@develop
​​​​spack develop py-islpy@develop
​​​​spack develop py-petsc4py@develop
​​​​spack develop py-pyadjoint@develop
​​​​spack develop py-pyop2@develop
​​​​spack develop py-coffee@develop
​​​​spack develop py-loopy@develop
​​​​spack develop py-cgen@develop

​​​​spack develop py-codepy@develop
​​​​spack develop py-genpy@develop
​​​​spack develop py-tsfc@develop
​​​​spack develop py-ufl@develop

  6. Add Firedrake:

    ​​​​spack add py-firedrake@develop \
​​​​    %gcc@9.3.0 \
​​​​    ^python@3.8.5.1 \
​​​​    ^cray-mpich@7.7.17%gcc@9.3.0 \
​​​​    ^cray-libsci@20.09.1 \
​​​​    ^firedrake.petsc@develop+fftw cflags=\"-O3 -march=native -mtune=native\" cxxflags=\"-O3 -march=native -mtune=native\" fflags=\"-O3 -march=native -mtune=native -ffree-line-length-512\"

  7. Add to $SPACK_ENV/spack.yaml:

    ​​​​spack:
​​​​  packages:
​​​​    all:
​​​​      compiler: [gcc@9.3.0]
​​​​      providers:
​​​​        mpi: [cray-mpich]
​​​​        blas: [cray-libsci]
​​​​        lapack: [cray-libsci]
​​​​        scalapack: [cray-libsci]
​​​​    netlib-scalapack:
​​​​      externals:
​​​​        # Pretend cray-libsci is netlib-scalapack
​​​​        - spec: netlib-scalapack@20.09.1
​​​​          modules:
​​​​          - cray-libsci/20.09.1
​​​​      buildable: False

  8. Concretize (and log) spack concretize -f 2>&1 | tee $SPACK_ENV/conc.log

    • Splitting the concretize and install steps into 2 distinct parts allows you to review what is being installed before the lengthy installation process begins.

  9. Install (and log) spack install --fail-fast 2>&1 | tee $SPACK_ENV/spack-firedrake-install.log

Testing

Testing must be run on a compute node. An interactive session can be started using

qsub -I -q arm-dev -l walltime=00:10:00

Alternatively submit a jobscript, see additional notes.

  1. Test you can import Firedrake by running python -c "from firedrake import *"
    • If this fails, before trying anything else, deactivate the environment with spack env deactivate and reactivate with spack env activate -p ./firedrake (as above) and try running python -c "from firedrake import *" again. This appears to be a shortcoming of spack.
  2. Run the basic functionality tests: 
    ​​​​cd $SPACK_ENV/py-firedrake
​​​​pytest tests/regression/ -m "not parallel" -k "poisson_strong or stokes_mini or dg_advection"
    • When running the environment on the compute nodes the variable $HOME needs to be set to something like /work/_projectcode_/_projectcode_/_username_ or /tmp as many dependencies expect to find configuration in $HOME.
    • We don't run parallel tests due to an issue with the current parallel pytest hook, this is in the process of being fixed.
  3. Run the full test suite: 
    ​​​​cd $SPACK_ENV/src/firedrake
​​​​pytest tests -m "not parallel"

Additional notes

Isambard maintains an installation of Spack, but it is not up to date and does not contain many of the fixes required to install Firedrake. Documentation is available on the Isambard spack docs page.

Select a repo