# Spack install Firedrake on Isambard
###### tags: `spack`

We have only tried installing on the XCI (ARM/ThunderX2/TX2) system, not the MACS, A64FX or Phase3 portions of the machine.

## Prerequisites
You should familiarise yourself with as much of Spack's functionality as possible. To achieve the same flexibility as we have on desktop installs the Firedrake Spack install leverages a lot of Spack's features. More information about Spack can be found on their [website](https://spack.readthedocs.io/en/latest/index.html).

## Spack
### Environment
- The following clones the Spack repo, loads modules to setup the environment and activates Spack.

git clone -c feature.manyFiles=true [e-mail]:spack/spack.git

module purge
module load Base-opts
module load PrgEnv-gnu
module load pmi-lib
module load cray-python/3.8.5.1
module load cray-mpich

. spack/share/spack/setup-env.sh

### Configuration
1. Populate `$SPACK_USER_CONFIG_PATH/cray/compilers.yaml` with
<details>
<summary>
<code>$SPACK_USER_CONFIG_PATH/cray/compilers.yaml</code>
</summary>

compilers:
- compiler:
    spec: cce@11.0.4
    paths:
      cc: cc
      cxx: CC
      f77: ftn
      fc: ftn
    flags: {}
    operating_system: cnl7
    target: any
    modules:
    - PrgEnv-cray
    - cce/11.0.4
    environment: {}
    extra_rpaths: []
- compiler:
    spec: gcc@9.3.0
    paths:
      cc: cc
      cxx: CC
      f77: ftn
      fc: ftn
    flags: {}
    operating_system: cnl7
    target: any
    modules:
    - PrgEnv-gnu
    - gcc/9.3.0
    environment: {}
    extra_rpaths: []

</details>

1. Populate `$SPACK_USER_CONFIG_PATH/packages.yaml` with
<details>
<summary>
<code>$SPACK_USER_CONFIG_PATH/packages.yaml</code>
</summary>

packages:
  all:
    providers:
      mpi: [cray-mpich]
  pkg-config:
    externals:
    - spec: pkg-config@0.29.2
      prefix: /usr
    buildable: False
  pkgconf:
    externals:
    - spec: pkgconf@0.29.2
      prefix: /usr
    buildable: False
  cray-mpich:
    externals:
    - spec: cray-mpich@7.7.17%gcc@9.3.0
      modules:
      - PrgEnv-gnu
      - gcc/9.3.0
      - cray-mpich/7.7.17
    - spec: cray-mpich@7.7.17%cce@11.0.4
      modules:
      - PrgEnv-cray
      - cce/11.0.4
      - cray-mpich/7.7.17
    buildable: False
  cray-libsci:
    externals:
      - spec: cray-libsci@20.09.1
        modules:
        - cray-libsci/20.09.1
      - spec: cray-libsci@19.06.1
        modules:
        - cray-libsci/19.06.1
    buildable: False
  python:
    externals:
      - spec: python@3.8.5.1
        modules:
        - cray-python/3.8.5.1
      - spec: cray-python@3.7.3.2
        modules:
        - cray-python/3.7.3.2
    buildable: False

</details>

4. Esure `$SPACK_USER_CONFIG_PATH/upstreams.yaml` does not exist. Upstream (system level installed Spack packages) for Isambard cause the Cray tools (compiler wrappers) to break.

### Firedrake
1. Clone the repo:

    git clone https://github.com/firedrakeproject/firedrake-spack.git
    # or
    git clone [e-mail]:firedrakeproject/firedrake-spack.git

1. Add the repository to spack `spack repo add <repo directory> `
1. Create an spack environment `spack env create -d ./firedrake`
1. Activate that environment `spack env activate -p ./firedrake`
1. To avoid a bunch of errors add a whole bunch of packages to the development package list:

    spack develop py-firedrake@develop
    spack develop libsupermesh@develop
    spack develop petsc@develop
    spack develop chaco@petsc
    spack develop py-fiat@develop
    spack develop py-finat@develop
    spack develop py-islpy@develop
    spack develop py-petsc4py@develop
    spack develop py-pyadjoint@develop
    spack develop py-pyop2@develop
    spack develop py-coffee@develop
    spack develop py-loopy@develop
    spack develop py-cgen@develop

    spack develop py-codepy@develop
    spack develop py-genpy@develop
    spack develop py-tsfc@develop
    spack develop py-ufl@develop

1. Add Firedrake:

    spack add py-firedrake@develop \
        %gcc@9.3.0 \
        ^python@3.8.5.1 \
        ^cray-mpich@7.7.17%gcc@9.3.0 \
        ^cray-libsci@20.09.1 \
        ^firedrake.petsc@develop+fftw cflags=\"-O3 -march=native -mtune=native\" cxxflags=\"-O3 -march=native -mtune=native\" fflags=\"-O3 -march=native -mtune=native -ffree-line-length-512\"

1. Add to `$SPACK_ENV/spack.yaml`:

    spack:
      packages:
        all:
          compiler: [gcc@9.3.0]
          providers:
            mpi: [cray-mpich]
            blas: [cray-libsci]
            lapack: [cray-libsci]
            scalapack: [cray-libsci]
        netlib-scalapack:
          externals:
            # Pretend cray-libsci is netlib-scalapack
            - spec: netlib-scalapack@20.09.1
              modules:
              - cray-libsci/20.09.1
          buildable: False

1. Concretize (and log) `spack concretize -f 2>&1 | tee $SPACK_ENV/conc.log`
* Splitting the concretize and install steps into 2 distinct parts allows you to review what is being installed before the lengthy installation process begins.

1. Install (and log) `spack install --fail-fast 2>&1 | tee $SPACK_ENV/spack-firedrake-install.log`

### Testing
Testing must be run on a compute node. An interactive session can be started using

qsub -I -q arm-dev -l walltime=00:10:00

Alternatively submit a jobscript, see [additional notes](#Additional-notes).
1. Test you can import Firedrake by running `python -c "from firedrake import *"`
- If this fails, before trying anything else, deactivate the environment with `spack env deactivate` and reactivate with `spack env activate -p ./firedrake` (as above) and try running `python -c "from firedrake import *"` again. This appears to be a shortcoming of spack.
1. Run the basic functionality tests:

    cd $SPACK_ENV/py-firedrake
    pytest tests/regression/ -m "not parallel" -k "poisson_strong or stokes_mini or dg_advection"

- When running the environment on the compute nodes the variable `$HOME` needs to be set to something like `/work/_projectcode_/_projectcode_/_username_` or `/tmp` as many dependencies expect to find configuration in `$HOME`.
- We don't run parallel tests due to an issue with the current parallel pytest hook, this is in the process of being fixed.
1. Run the full test suite:

    cd $SPACK_ENV/src/firedrake
    pytest tests -m "not parallel"

## Additional notes
Isambard maintains an installation of Spack, but it is not up to date and does not contain many of the fixes required to install Firedrake. Documentation is available on the [Isambard spack docs](https://gw4-isambard.github.io/docs/user-guide/spack.html) page.