# Implementation of ICON stencils

###### tags: `functional cycle 9`

Developer: Nikki, Samuel, Erwan, Hannes, other once free

# ICON Stencil Integration

## Process for integration

All dycore stencils have been implemented in dusk and are available [here](https://github.com/C2SM/icon-exclaim/tree/icon-dsl/dsl).

First dusk stencils must be ported to gt4py and merged into [`icon4py`](https://github.com/C2SM/icon4py). To port from dusk to Gt4Py the stencil should be implemented as a `field_operator` and called from a `program`. Furthermore a corresponding unit test with a reference implementation in `numpy` should also be available for each stencil.

To integrate stencils into `icon-exclaim` the following needs to be carried out:

- Get access to `tsa` machine, and in `/scratch/<user>` execute the setup script located [here](https://github.com/C2SM/icon-exclaim/blob/add_gt4py_codegen/dsl/setup.sh). This script will download all dependencies, setup the local environment and compile icon.

- For each stencil to be integrate two files must be modified. The `CMakeLists.txt` file, which enables control over whether to use dusk or gt4py for a given stencil, and the corresponding `f90` source code containining the fortran wrapper subroutine which calls generated gt4py code.

- Note that when editing the wrapper subroutine depending on the execution domain of the stencil two or four extra keyword arguments have to be added, namely `vertical_lower, vertical_upper, horizontal_lower, horizontal_upper`. The corresponding variable for each keyword argument can be found by inspecting the Fortran loop index variables.

- Once the corresponding changes have been made ICON needs to be recompiled and a simulation run in `verification` mode, which runs the integrated stencils and the original fortran stencils side by side and compares their results (see [README](https://github.com/C2SM/icon-exclaim/tree/add_gt4py_codegen/dsl) for commands, serialising the data if the error is above a certain threshold (e-06).

- To inspect whether the stencils ran without errors you can inspect the `LOG` file generated under `icon-exclaim/build_debug/run` for run `mch_ch_r04b09`. If no errors are present a PR can be opened into `icon-exclaim` and the full jenkins CI/CD pipeline can be triggered by adding a comment: `launch jenkins`. This will recompile icon and run in verification and substitution mode, whilst also running a probtest.

## Integrated

### mo_nh_diffusion_stencil_10.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_19
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_43.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_11.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_66.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_39.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_40.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_05.py
- [X] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_06.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_08.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_09.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_48.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_velocity_advection_stencil_02.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_velocity_advection_stencil_03.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_nh_diffusion_stencil_03.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_07.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_54.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Merged into icon-exclaim

### mo_solve_nonhydro_stencil_31.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_61.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_velocity_advection_stencil_05.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_17.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Merged into icon-exclaim

### mo_velocity_advection_stencil_14.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_15.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_33.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_41.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_44.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_46.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_67.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_37.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_nh_diffusion_stencil_08.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_32.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_02.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_nh_diffusion_stencil_02.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_velocity_advection_stencil_18.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_nh_diffusion_stencil_05.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_nh_diffusion_stencil_12.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_nh_diffusion_stencil_09.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_nh_diffusion_stencil_07.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_25.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_velocity_advection_stencil_07.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Merged into icon-exclaim

### mo_solve_nonhydro_stencil_18.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Merged into icon-exclaim

### mo_nh_diffusion_stencil_13.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Merged into icon-exclaim

### mo_nh_diffusion_stencil_14.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_velocity_advection_stencil_01.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_nh_diffusion_stencil_06.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_velocity_advection_stencil_11.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_velocity_advection_stencil_04.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_60.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_65.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_24.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_23.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_26.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_35.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_59.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_29.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_58.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_velocity_advection_stencil_13.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_62.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [X] Ported and verified in `icon-exclaim`

### mo_velocity_advection_stencil_12.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_45.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_03.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_57.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_64.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_velocity_advection_stencil_08.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_velocity_advection_stencil_09.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_velocity_advection_stencil_17.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_nh_diffusion_stencil_11.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [X] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_01.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_30.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_34.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_42.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_10.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_12.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_36.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_49.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_velocity_advection_stencil_06.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

**Note :** with an incorrect choice of vertical bounds, it throws an illegal memory access error.

### mo_velocity_advection_stencil_15.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_velocity_advection_stencil_20.py - Hannes/BW
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_04.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_55.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_56.py
- [X] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_63.py
- [X] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_nh_diffusion_stencil_16.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_velocity_advection_stencil_16.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_51.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_27.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

*Stencil is reached in present experiment if the `divdamporder` namelist option is set to 4 (from 24)*

### mo_solve_nonhydro_stencil_13.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_14.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_50.py (not run in simulation)
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_28.py (not run in simulation)
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_38.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_68.py (bool mask) - BW
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_math_divrot_rot_vertex_ri_dsl.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_icon_interpolation_scalar_cells2verts_scalar_ri_dsl.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_intp_rbf_rbf_vec_interpol_vertex.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_math_gradients_grad_green_gauss_cell_dsl.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_nh_diffusion_stencil_01.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_velocity_advection_stencil_19.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_16.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_nh_diffusion_stencil_04.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_47.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

*fixed by setting nlevp1 for vertical bounds*

### mo_velocity_advection_stencil_10.py
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_22.py (bool mask)
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [x] Ported and verified in `icon-exclaim`

**TYPE mismatch in boolean mask between Fortran mask and Fortran interface**

## Stencils with integration issues

### mo_solve_nonhydro_stencil_20.py
- [x] Translated to `gt4py`
- [ ] Merged into `icon4py`
- [ ] Ported and verified in `icon-exclaim`

**weights on sparse field, IndexField**

### mo_solve_nonhydro_stencil_21.py
- [x] Translated to `gt4py`
- [ ] Merged into `icon4py`
- [ ] Ported and verified in `icon-exclaim`

**weights on sparse field, IndexField**

### mo_nh_diffusion_stencil_15.py
- [x] Translated to `gt4py`
- [ ] Merged into `icon4py`
- [ ] Ported and verified in `icon-exclaim`

**IndexField, vertical indirection, sparse field with k dimension**

NVFORTRAN-S-0079-Keyword form of argument illegal in this context for ikidx (/scratch/skellerh/icon-exclaim/src/atm_dyn_iconam/mo_solve_nonhydro.f90: 1899)
  0 inform,   0 warnings,   1 severes, 0 fatal for solve_nh
get_ptr_to_var_data:

NVFORTRAN-W-0435-Array declared with zero size (/scratch/skellerh/icon-exclaim/src/atm_phy_echam/mo_interface_echam_cnv.f90: 75)
NVFORTRAN-W-0435-Array declared with zero size (/scratch/skellerh/icon-exclaim/src/atm_phy_echam/mo_interface_echam_cnv.f90: 93)

---

## Integratable/Translatable stencils (In Progress)

**Scan operator**

### mo_solve_nonhydro_stencil_52.py - NF
- [x] Translated to `gt4py`
- [ ] Merged into `icon4py`
- [ ] Ported and verified in `icon-exclaim`

### mo_solve_nonhydro_stencil_53.py - Hannes
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [ ] Ported and verified in `icon-exclaim`

### mo_advection_traj_btraj_compute_o1_dsl.py, weights on sparse field - AG
- [x] Translated to `gt4py`
- [x] Merged into `icon4py`
- [ ] Ported and verified in `icon-exclaim`

## Integration Issues

**TYPE mismatch in boolean mask between Fortran mask and Fortran interface**

In this case Fortran does not provide a boolean mask, instead it provides a field of doubles as that is how it was handled in dusk. In gt4py we need a field of bools.

**How do we port and integrate stencils which make use of an `IndexField`**

### Resolved:
- Weights on a sparse field
- KIND mismatch in boolean mask between Fortran mask and Fortran interface
- stencils with multiple returns
- fields with only kdim
- origin offsets not found in generated
- icon4pygen parse fields that are both input and output fields.
- Stencils with backwards loops.
- Stencils with only one forward loop (missing vertical dimension).
- icon4pygen parse multi-node PASTs.
- return float(0.0): issue was with the fact that there were no inputs fields while gt4py was looking for a "," to parse the inputs
- generating correct number of neighbors for neighbor chains using icochainsize.
- avoid function namespace clashing by inlining.
- Fortran LOGICAL/boolean integration

# Dusk to Gt4Py porting

## Currently not portable stencils