# icon-dsl with serialization build on balfrin

Below is a guide of how to compile icon-dsl on balfrin with serialization. Currently, serialbox library is not in the user environment. You have to install it manually.

## Before you start

For the build described in this hackmd, you need to have `dsl/env.sh`, `dsl/setup.sh`, and `config/cscs/alps_nospack`.
They were introduced in this PR: https://github.com/C2SM/icon-exclaim/pull/283 so until they are not merged in main, borrow them from there.

## Steps

1. Prepare the uenv image. Details about uenv can be found on https://confluence.cscs.ch/display/KB/UENV+user+environments

git clone https://github.com/eth-cscs/uenv.git
cd uenv
# run the installation script.
# if this is the first time that you are installing uenv,
# type "yes" or "y" when prompted whether to update your ~/.bashrc file.
./install --local

2. activate the environment (you need this also for running ICON, so put it in your run scripts or bashrc)

uenv start --view=icon-wcp:icon /scratch/mch/leclairm/uenvs/images/icon.v1.rc4.sqfs

You should see something as shown below when you type ```uenv status```

/user-environment:icon-wcp
  The tools required to build ICON
  views:
    icon (loaded)
    modules: activate modules
    spack: configure spack upstream

3. clone the repo and move to the branch you need to work with

git clone [e-mail]:C2SM/icon-exclaim.git
cd icon-exclaim
git submodule update --init --recursive

4. Prepare the serialbox library

4.1. **Spack**

git clone --depth 1 --recurse-submodules --shallow-submodules -b v0.21.1.3 https://github.com/C2SM/spack-c2sm.git
cd spack-c2sm
source setup-env.sh # this line should actually be `source setup-env.sh /user-environment`, but neither Chia Rui nor Jacopo have tested it
spack install serialbox%nvhpc@24.5 +fortran # this will take quite some time

if spack does not automatically detect the compiler, you can do the following to find the correct compiler and rerun the command above

spack compiler find

In the script ```setup.sh```, you need to add the following lines

source /scratch/mch/<your-user-name>/spack-c2sm/setup-env.sh # wherever this is
spack load serialbox%nvhpc@24.5

and in `env.sh` change this variable

SERIALBOX2_ROOT=$(spack find --format "{prefix}" serialbox@2.6.1%nvhpc@24.5 | head -n1)

4.2. **No spack**
Or, if you do not want to use spack, you can manually install serialbox by (which is much faster than spack install above)

git clone https://github.com/eth-cscs/serialbox2.git
cd serialbox2/
mkdir build
cd build/
export CC=mpicc
export FC=mpif90
export CXX=mpicxx
cmake ../ -DSERIALBOX_ENABLE_FORTRAN=ON
cmake --build .
cmake --build . --target install

Now you have to change `SERIALBOX2_ROOT` to your serialbox installation path in the script `env.sh`, for example,

SERIALBOX2_ROOT=/scratch/mch/cong/serialbox2/install

5. Build icon-dsl.

Building on cpu:
In `setup.sh`:
- add an option for a new `build_type` at the top: `build_serialize_cpu`;
- in `build_for_type()` copy the build_cpu options as follows

build_for_type() {
    local build_type=$1
    if [[ "$build_type" == "build_cpu" || "$build_type" == "build_serialize_cpu" ]]; then
        build_dir="build_cpu"
            gpu_option=""
        liskov_option="--enable-dsl-local"
        ACC_FC_FLAGS=""
        CUDA_LDFLAGS=""
        CUDA_LIBS=""
        NVCFLAGS=""
        if [ "$build_type" == "build_serialize_cpu" ]; then
            build_dir="build_serialize_cpu"
            liskov_option="--enable-serialization"
        fi   

Finally, copy `alps_nospack` to `icon-exclaim/dsl/icon-exclaim/config/cscs/`. Then, run the `setup.sh` script (note that the setup script and `env.sh` must be at the same directory with icon-exclaim, icon4py, and gridtools)

./setup.sh build_serialize

## Troubleshooting:

a. If you encounter this error:

NVFORTRAN-F-0004-Unable to open MODULE file mo_nonhydro_state.mod (/scratch/mch/jcanton/icon-exclaim/icon-exclaim/src/shr_horizontal/mo_icon_interpolation_scalar.f90: 35)
NVFORTRAN/x86-64 Linux 24.5-1: compilation aborted

you need to remove the serialization code (lines with `!$ser`) from `mo_icon_interpolation_scalar.f90`.

b. Similar as above:

make[1]: Entering directory '/scratch/mch/jcanton/icon-exclaim/icon-exclaim/build_serialize_cpu'
  FC       src/shr_horizontal/mo_math_gradients.pp-fpp.pp-sb2.o
NVFORTRAN-F-0004-Unable to open MODULE file mo_nonhydro_state.mod (/scratch/mch/jcanton/icon-exclaim/icon-exclaim/src/shr_horizontal/mo_math_gradients.f90: 45)
NVFORTRAN/x86-64 Linux 24.5-1: compilation aborted

remove serialization code in `mo_math_gradients.f90`