---
# System prepended metadata

title: Install Seissol on Taiwania3 (Intel Node)

---

Install Seissol on Taiwania3 (Intel Node)
=============

### Module load GCC
```bash=
ml gcc/13
```

### Install Spack
```bash=
git clone https://github.com/spack/spack.git
. ~/spack/share/spack/setup-env.sh
```

### Download Dependencies for Seissol
```bash=
spack install seissol convergence_order=4 gemm_tools_list=LIBXSMM,OpenBLAS,PSpaMM graph_partitioning_libs=parmetis
```

### Fail on easi(110/111)
![image](https://hackmd.io/_uploads/HJrtO5ITkx.png)

This cmake would automatically check the compiler is work or not.
Ashole just can't fetch the fucking GCC.
=> Specify path for it.
=> Noted that we should specify the `openmpi` path for it. Or this would happen again when it starts to detect `MPI_C_COMPILER` ==

Edit the cached install package in `~/spack/var/spack/cache/_source-cache/git/SeisSol/easi.git`
```bash=
cp ~/spack/var/spack/cache/_source-cache/git/SeisSol/easi.git/0d87b1a7db31e453d52c7213cb9b31bda88cbf40.tar.gz ~
tar xvf 0d87b1a7db31e453d52c7213cb9b31bda88cbf40.tar.gz
```
After decompress it, you should get a `easi` directory on your `~`

Append this two line into the CmakeList.txt
(`~/easi/CMakeLists.txt`)
```bash=
set(CMAKE_C_COMPILER "~/spack/opt/spack/linux-cascadelake/openmpi-5.0.7-eee5iqsn5ncps7x3jnlf64334v5mq4yg/bin/mpicc")
set(CMAKE_CXX_COMPILER "~/spack/opt/spack/linux-cascadelake/openmpi-5.0.7-eee5iqsn5ncps7x3jnlf64334v5mq4yg/bin/mpic++")
```
Like this:
![image](https://hackmd.io/_uploads/HyGuFcUTJx.png)

Finally, tar this modified `~/easi` directory back to the same name as the .tar.gz in spack cache.

```bash=
tar -cvf 0d87b1a7db31e453d52c7213cb9b31bda88cbf40.tar.gz easi
```

Drag this .tar.gz back and replace the original one.
![image](https://hackmd.io/_uploads/Sy0T5qUayl.png)

OK, now we deal with this ashole `easi` again.
```bash=
spack install seissol convergence_order=4 gemm_tools_list=LIBXSMM,OpenBLAS,PSpaMM graph_partitioning_libs=parmetis
```

This time it should be OK.

### Fail on Seissol(111/111)

Then this should failed, bitch ==
But not a big deal.
We could have it manually.

### Build Seissol from source

```bash=
git clone https://github.com/SeisSol/SeisSol.git
mv SeisSol seissol_tmp
cd seissol_tmp
git submodule update --init --recursive
mkdir build && cd build
```

### Spack load ...
```bash=
spack load gcc openmpi cmake eigen easi hdf5 \
netcdf-c py-pspamm openblas openmpi parmetis \
yaml-cpp libxsmm
```
### Add netcdf-c into LIBRARY_PATH and LD_LIBRARY_PATH
```bash=
export LD_LIBRARY_PATH="/home/jason3410/spack/opt/spack/linux-cascadelake/netcdf-c-4.9.2-bi6acrct4tzme7nyxagyj4udl6erfzzt/lib:$LD_LIBRARY_PATH"
export LIBRARY_PATH="/home/jason3410/spack/opt/spack/linux-cascadelake/netcdf-c-4.9.2-bi6acrct4tzme7nyxagyj4udl6erfzzt/lib:$LIBRARY_PATH"
export C_INCLUDE_PATH="/home/jason3410/spack/opt/spack/linux-cascadelake/netcdf-c-4.9.2-bi6acrct4tzme7nyxagyj4udl6erfzzt/include:$C_INCLUDE_PATH"
export CPLUS_INCLUDE_PATH="/home/jason3410/spack/opt/spack/linux-cascadelake/netcdf-c-4.9.2-bi6acrct4tzme7nyxagyj4udl6erfzzt/include:$CPLUS_INCLUDE_PATH"
```

### Python Dependencies
```bash=
python -m ensurepip --upgrade
pip3 install --upgrade pip

python -m pip install numpy
python -m pip install pkg_resources
python -m pip install setuptools
python -m pip install scipy
```


### Install
```bash=

#hsw: haswell, supports AVX2
cmake -DCMAKE_BUILD_TYPE=Release \
      -DNUMA_AWARE_PINNING=ON \
      -DASAGI=ON \
      -DPRECISION=double \
      -DORDER=4 \
      -DEQUATIONS=elastic \
      -DGRAPH_PARTITIONING_LIBS=parmetis \
      -DCMAKE_PREFIX_PATH="~/spack/opt/spack/linux-cascadelake/netcdf-c-4.9.2-bi6acrct4tzme7nyxagyj4udl6erfzzt/lib
      ..
      
make -j$(nproc)
```
### Update Linker
Another shitty error here is:
![image](https://hackmd.io/_uploads/S1tRp9Ipkx.png)

```bash=
/usr/bin/ld: /home/jason3410/spack/opt/spack/linux-cascadelake/libxsmm-1.17-mzyyr5lb6rvpo4jywp6ckziho5wwilah/lib/libxsmm.a(libxsmm_main.o): unable to initialize decompress status for section .debug_info
/usr/bin/ld: /home/jason3410/spack/opt/spack/linux-cascadelake/libxsmm-1.17-mzyyr5lb6rvpo4jywp6ckziho5wwilah/lib/libxsmm.a(libxsmm_main.o): unable to initialize decompress status for section .debug_info
/home/jason3410/spack/opt/spack/linux-cascadelake/libxsmm-1.17-mzyyr5lb6rvpo4jywp6ckziho5wwilah/lib/libxsmm.a: error adding symbols: File format not recognized
collect2: error: ld returned 1 exit status
```

It is due the the out-of-date linker version.
![image](https://hackmd.io/_uploads/ry-UC5Ipkg.png)


To solve this, we install latest version of linker with spack.
```bash=
spack install binutils
spack load binutils
```
Now:
![image](https://hackmd.io/_uploads/BylKAcIpyx.png)

### Done
Finally, we could finish this.
```bash=
make -j$(nproc)
```
![image](https://hackmd.io/_uploads/ByoiC9UT1g.png)


### Testing
```bash=
[jason3410@cpn3006 build]$ ~/seissol/build/SeisSol_proxy_Release_dhsw_4_elastic 100000 100 all
Allocating fake data... ...done
=================================================
===            PERFORMANCE SUMMARY            ===
=================================================
seissol proxy mode                  : all
time for seissol proxy              : 2.51268
cycles                              : 0

GFLOP (libxsmm)                     : 0
GFLOP (pspamm)                      : 0
GFLOP (libxsmm + pspamm)            : 0
GFLOP (non-zero) for seissol proxy  : 507.137
GFLOP (hardware) for seissol proxy  : 867.065
GiB (estimate) for seissol proxy    : 193.119

FLOPS/cycle (non-zero)              : inf
FLOPS/cycle (hardware)              : inf
Bytes/cycle (estimate)              : inf

GFLOPS (non-zero) for seissol proxy : 201.831
GFLOPS (hardware) for seissol proxy : 345.075
GiB/s (estimate) for seissol proxy  : 76.8577
=================================================
```

### Env-setup.<span>sh (for future usage)
```bash=
#! /bin/bash
ml gcc/13
spack load gcc openmpi cmake eigen easi hdf5 \
netcdf-c py-pspamm openblas openmpi parmetis \
yaml-cpp libxsmm binutils python

export LD_LIBRARY_PATH="/home/jason3410/spack/opt/spack/linux-cascadelake/netcdf-c-4.9.2-bi6acrct4tzme7nyxagyj4udl6erfzzt/lib:$LD_LIBRARY_PATH"
```