# Notes on new Firedrake Spack installer
###### tags: `spack`
## Instructions:
### [Spack](https://spack.readthedocs.io/en/latest/index.html):
1. Follow [getting started](https://spack.readthedocs.io/en/latest/getting_started.html) guide
- Ensure develop branch is installed
- Ensure desired [compilers](https://spack.readthedocs.io/en/latest/getting_started.html#compiler-configuration) are known to spack, should be picked up by `spack compiler find`
- Ensure [system packages](https://spack.readthedocs.io/en/latest/getting_started.html#system-packages) (that you don't want spack to install) are known to spack, some may be picked up by `spack external find`, others may require manual intervention.
2. You may want to specify the system MPI as the correct MPI for spack to use. This appears to be quite difficult, a starting template for the `~/.spack/packages.yaml`might be:

    packages:
      mpich:
      externals:
      - spec: mpich@4.0
        prefix: /opt/mpich
      buildable: False

If Spack's PETSc package doesn't think this spec line is suitable it _will_ try to build its own MPICH.

Make sure the spack `setup-env.sh` file has been sourced and you can run spack commands before moving to the next section.

### [Firedrake-repo](https://github.com/firedrakeproject/firedrake-spack)
1. Clone the above repo
1. Add the repository to spack `spack repo add <repo directory> `
1. Create an spack environment `spack env create -d ./firedrake`
1. Activate that environment `spack env activate -p ./firedrake`
* To avoid a bunch of errors add a whole bunch of packages to the development package list:

    spack develop py-firedrake@develop
    spack develop libsupermesh@develop
    spack develop petsc@develop
    spack develop chaco@petsc
    spack develop py-fiat@develop
    spack develop py-finat@develop
    spack develop py-islpy@develop
    spack develop py-petsc4py@develop
    spack develop py-pyadjoint@develop
    spack develop py-pyop2@develop
    spack develop py-coffee@develop
    spack develop py-loopy@develop
    spack develop py-cgen@develop
    
    spack develop py-codepy@develop
    spack develop py-genpy@develop
    spack develop py-tsfc@develop
    spack develop py-ufl@develop

*
1. Install firedrake using `spack add py-firedrake@develop %gcc ^mpich ^openblas`
* `%gcc` specifies the compiler, you may wish to specify a version (eg: `%gcc@11.2.0`), ommiting this will use the default compiler
* `^mpich` specifies which MPI to use. If you set an MPI system package be sure to use this (eg: `^openmpi@3.1`)
* `^openblas` specifies which BLAS/LAPACK library to use. If you set an BLAS/LAPACK system package be sure to use this (eg: `^intel-mkl@2019.1`)
* You can further specify the Python version by adding `^python@3.10` for Python 3.10 or setting the system python if it's set up in system packages
1. `spack install` Will install everything
- Add `--fail-fast` to stop at the first package with an install error
- Adding `2>&1 | tee spack-firedrake-install.log` to the end of the command will save all output to a log file that you can send if you need someone else to look over the output.
- ```spack install --fail-fast 2>&1 | tee spack-firedrake-install.log ```
1. Test you can import Firedrake by running `python -c "from firedrake import *"`
- If this fails, before trying anything else, deactivate the environment with `spack env deactivate` and reactivate with `spack env activate -p ./firedrake` (as above) and try running `python -c "from firedrake import *"` again. This appears to be a shortcoming of spack (related to[#10801](https://github.com/spack/spack/issues/10801)?).
1. Run the basic functionality tests:

    cd $SPACK_ENV/py-firedrake
    pytest tests/regression/ -k "poisson_strong or stokes_mini or dg_advection"

1. Run the full test suite:

    cd $SPACK_ENV/src/firedrake
    pytest tests

## Will's Notes:

## Yohance's Notes:

## Connor's Notes:

**26/04** It works! Great job.

> - I didn't manage to use the system MPICH. Some information on how to configure that would be excellent.
> - > Turns out this is quite hard to do, not in general, but for our use case. PETSc seems to have a fairly strict spec for how it installs MPICH, if the system spec isn't specified the same way spack will go ahead and install its own. You need in depth knowledge of how your MPI was built to specify itg as an external package, apparently. I have added something above to start from at least.
> - This isn't quite working for me at the moment. I run the installer and everything appears to work fine but then I run `python -c "import firedrake"` and I get
> ```bash
> $ python -c "import firedrake"
> Traceback (most recent call last):
> File "<string>", line 1, in <module>
> File "/home/connor/Code/firedrake-spack/firedrake/py-firedrake/firedrake/__init__.py", line 69, in <module>
> from pyop2 import op2 # noqa: F401
> ModuleNotFoundError: No module named 'pyop2'
> ```
> Importing `petsc4py`, `ufl`, `tsfc` all work perfectly well.
>
> I have tried going to `py-pyop2`, running `git pull` and then `spack install --verbose`. The relevant part of the output is:
> ```bash
> ==> Installing py-pyop2-develop-sgqienmyqamtxbnrucc4mcy3mg2xecyo
> ==> No binary for py-pyop2-develop-sgqienmyqamtxbnrucc4mcy3mg2xecyo found: installing from source
> ==> No patches needed for py-pyop2
> ==> py-pyop2: Executing phase: 'install'
> ==> [2022-04-04-15:51:07.079513] '/home/connor/.local/opt/spack/opt/spack/linux-arch-haswell/gcc-11.1.0/python-3.9.10-m2mnygxubd43ebkhdp5kgbjp3q5fkax5/bin/python' 'setup.py' 'develop' '--prefix=/home/connor/.local/opt/spack/opt/spack/linux-arch-haswell/gcc-11.1.0/py-pyop2-develop-sgqienmyqamtxbnrucc4mcy3mg2xecyo'
> /home/connor/.local/opt/spack/opt/spack/linux-arch-haswell/gcc-11.1.0/python-3.9.10-m2mnygxubd43ebkhdp5kgbjp3q5fkax5/lib/python3.9/distutils/dist.py:274: UserWarning: Unknown distribution option: 'test_requires'
> warnings.warn(msg)
> running develop
> /home/connor/.local/opt/spack/opt/spack/linux-arch-haswell/gcc-11.1.0/py-setuptools-59.4.0-zze3l2hthtttpbdcafr3s4qthvcjfmnl/lib/python3.9/site-packages/setuptools/command/easy_install.py:156: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
> warnings.warn(
> /home/connor/.local/opt/spack/opt/spack/linux-arch-haswell/gcc-11.1.0/py-setuptools-59.4.0-zze3l2hthtttpbdcafr3s4qthvcjfmnl/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
> warnings.warn(
> running egg_info
> writing PyOP2.egg-info/PKG-INFO
> writing dependency_links to PyOP2.egg-info/dependency_links.txt
> writing requirements to PyOP2.egg-info/requires.txt
> writing top-level names to PyOP2.egg-info/top_level.txt
> reading manifest file 'PyOP2.egg-info/SOURCES.txt'
> reading manifest template 'MANIFEST.in'
> adding license file 'LICENSE'
> adding license file 'AUTHORS'
> writing manifest file 'PyOP2.egg-info/SOURCES.txt'
> running build_ext
> skipping 'pyop2/sparsity.c' Cython extension (up-to-date)
> Creating /home/connor/.local/opt/spack/opt/spack/linux-arch-haswell/gcc-11.1.0/py-pyop2-develop-sgqienmyqamtxbnrucc4mcy3mg2xecyo/lib/python3.9/site-packages/PyOP2.egg-link (link to .)
> Adding PyOP2 0.13.0+637.g5f4c036b to easy-install.pth file
> Installing spydump script to /home/connor/.local/opt/spack/opt/spack/linux-arch-haswell/gcc-11.1.0/py-pyop2-develop-sgqienmyqamtxbnrucc4mcy3mg2xecyo/bin
> Installing pyop2-clean script to /home/connor/.local/opt/spack/opt/spack/linux-arch-haswell/gcc-11.1.0/py-pyop2-develop-sgqienmyqamtxbnrucc4mcy3mg2xecyo/bin
>
> Installed /home/connor/Code/firedrake-spack/firedrake/py-pyop2
> /home/connor/.local/opt/spack/opt/spack/linux-arch-haswell/gcc-11.1.0/py-setuptools-59.4.0-zze3l2hthtttpbdcafr3s4qthvcjfmnl/lib/python3.9/site-packages/pkg_resources/__init__.py:116: PkgResourcesDeprecationWarning: dev is an invalid version and will not be supported in a future release
> warnings.warn(
> /home/connor/.local/opt/spack/opt/spack/linux-arch-haswell/gcc-11.1.0/py-setuptools-59.4.0-zze3l2hthtttpbdcafr3s4qthvcjfmnl/lib/python3.9/site-packages/pkg_resources/__init__.py:116: PkgResourcesDeprecationWarning: is an invalid version and will not be supported in a future release
> warnings.warn(
> Processing dependencies for PyOP2==0.13.0+637.g5f4c036b
> Searching for networkx
> Reading https://pypi.org/simple/networkx/
> Downloading https://files.pythonhosted.org/packages/f6/34/4913f651b8e178dde5abcf8d62495e4dcd0757a9a6840f1b1f7a290afaea/networkx-2.7.1-py3-none-any.whl#sha256=011e85d277c89681e8fa661cf5ff0743443445049b0b68789ad55ef09340c6e0
> Best match: networkx 2.7.1
> Processing networkx-2.7.1-py3-none-any.whl
> Installing networkx-2.7.1-py3-none-any.whl to /home/connor/.local/opt/spack/opt/spack/linux-arch-haswell/gcc-11.1.0/py-pyop2-develop-sgqienmyqamtxbnrucc4mcy3mg2xecyo/lib/python3.9/site-packages
> Adding networkx 2.7.1 to easy-install.pth file
>
> Installed /home/connor/.local/opt/spack/opt/spack/linux-arch-haswell/gcc-11.1.0/py-pyop2-develop-sgqienmyqamtxbnrucc4mcy3mg2xecyo/lib/python3.9/site-packages/networkx-2.7.1-py3.9.egg
> Searching for COFFEE==0.1.0
> Best match: COFFEE 0.1.0
> Adding COFFEE 0.1.0 to easy-install.pth file
>
> Using /home/connor/.local/opt/spack/opt/spack/linux-arch-haswell/gcc-11.1.0/py-coffee-develop-3xt7ez35lt4h7pjlmjtokdzc5x6rylvs/lib/python3.9/site-packages
> Searching for numpy==1.22.3
> Best match: numpy 1.22.3
> Adding numpy 1.22.3 to easy-install.pth file
> Installing f2py script to /home/connor/.local/opt/spack/opt/spack/linux-arch-haswell/gcc-11.1.0/py-pyop2-develop-sgqienmyqamtxbnrucc4mcy3mg2xecyo/bin
> Installing f2py3 script to /home/connor/.local/opt/spack/opt/spack/linux-arch-haswell/gcc-11.1.0/py-pyop2-develop-sgqienmyqamtxbnrucc4mcy3mg2xecyo/bin
> Installing f2py3.9 script to /home/connor/.local/opt/spack/opt/spack/linux-arch-haswell/gcc-11.1.0/py-pyop2-develop-sgqienmyqamtxbnrucc4mcy3mg2xecyo/bin
>
> Using /home/connor/.local/opt/spack/opt/spack/linux-arch-haswell/gcc-11.1.0/py-numpy-1.22.3-ibokt6dedipcsv7xen66opjozhnte5el/lib/python3.9/site-packages
> Searching for mpi4py==3.1.2
> Best match: mpi4py 3.1.2
> Adding mpi4py 3.1.2 to easy-install.pth file
>
> Using /home/connor/.local/opt/spack/opt/spack/linux-arch-haswell/gcc-11.1.0/py-mpi4py-3.1.2-wswdc4g42s2ptl2i46adw3wxobtctt2h/lib/python3.9/site-packages
> Searching for decorator==4.4.2
> Best match: decorator 4.4.2
> Adding decorator 4.4.2 to easy-install.pth file
>
> Using /home/connor/.local/opt/spack/opt/spack/linux-arch-haswell/gcc-11.1.0/py-decorator-4.4.2-pk6eey7riaoevbowzmm5ojlk3dtlqd33/lib/python3.9/site-packages
> Finished processing dependencies for PyOP2==0.13.0+637.g5f4c036b
> ==> py-pyop2: Successfully installed py-pyop2-develop-sgqienmyqamtxbnrucc4mcy3mg2xecyo
> Fetch: 0.00s. Build: 1.63s. Total: 1.63s.
> ```
> This still fails to resolve the import problem even though everything seems to have gone through fine.
> > I think I now have a fix for this