# Finish Transition to GT4Py

###### tags: `functional cycle 13`

Developers: Matthias, Hannes
Appetite: until it's done

## Background

There are still 7 stencils that are not ported to gt4py (from dusk). Three of them are stencils written in `itir` that work on index fields, two are scan stencils, one is trivial (set-to-zero) and one is `mo_velocity_advection_stencil_18` that fails in integration for unknown reasons. The goal of this task is to complete the transition to gt4py (all stencil in gt4py, all stencils verifying, all stencils probtesting)

Additional 2 stencils will have to be ported because of global experiment (result of aquaplanet work) and another 2 stencils need to be re-integrated since they were inlined and manually merged in newest icon-nwp master used for aquaplanet.

**TODO** this might change after cooldown of 12 is done, potentially only 18 will be left

## Known Steps

- `mo_velocity_advection_stencil_18` will be challenging, previous attempts to make sense of the situation faild. One posible way forward might be to use cpu execution of `gtfn` and compile the stencil with the `gcc` address and undefined behavior sanitizers.
- Remove dusk/dawn

## Status

### mo_velocity_advection_stencil_18

- validates in verification, but not in probtest
- both imperative and functional don't work
~~- verification mode passes, but in the same run probtest doesn't pass with GT4Py (but does with dusk)~~
~~- probtest doesn't pass in both verification and substitution mode (MR TODO: recheck if holds true for verification)~~
- stencil18 is a noop, because all the masks are 0
- using probtest in verification mode now passes, did not pass in last attempt to make this work

Ideas:
- gt4py generated code touches memory that the verification doesn't check (but cuda-memcheck passes)?
- some setup routine have undefined behavior
- ~~does verification trigger openeacc memcopies in one case but not the other~~

Possible tests:
- run through profiler and check if there are weird instructions?
- write a standalone driver
- plain stencil without `where`, just `return ddt_w...` **This passes**
- starting from copy stencil and adding code from the "full" stencil 18 leads to a few versions that pass. The minimal version that _fails_ that was found is

    difcoef = where(
        levelmask,
        scalfac_exdiff * abs(z_w_con_c),
        0.0,
    )

    ddt_w_adv = where(
        levelmask,
        ddt_w_adv
        + difcoef * area * neighbor_sum(w(C2E2CO) * geofac_n2s, axis=C2E2CODim),
        ddt_w_adv,
    )

    return ddt_w_adv

Note: this is makes sense with the information learned below. The stencils started failing as soon as `difcoef != 0`, i.e. as soon as the stencil wasn't a conceptual no-op, leading to wrong values in case `levelmask` was uninitialized (and hence `true` for some levels)

### Update 02/02

Reverting the boolean fields (`LOGICAL` / `bool`) to integer fields (`INTEGER` / `int32`) and comparing against value `1` explicitly makes sten 18 work, c.f. [icon-exclaim PR](https://github.com/C2SM/icon-exclaim/pull/130), [icon4py PR](https://github.com/C2SM/icon4py/pull/151).

Some things of note **for previous attempt**:
* Previous attempts at integrating sten 18 used `levelmask=levmask(:,1)`, where `LOGICAL :: levmask(p_patch%nblks_c,p_patch%nlev)`. This means the `1` in the assignemt is misplaced, the block should be fixed, not the level.
* Fixing this, i.e. using `levelmask=levmask(1,:)` leads to a present table dump (?!). It is unclear why this would happen, or even matter, since `p_patch%nblks_c` is `1`.
* Replacing `LOGICAL :: levmask(p_patch%nblks_c,p_patch%nlev)` with `LOGICAL :: levmask(p_patch%nlev)` leads to a failing probtest (but verifies)

### Fixing the Bug

* The issue was that stencil 18 consumed two fields, `levmask` and `cfl_clipping`, that were only updated if `__DSL_VERIFY` is true
* This means, in substitution mode the `levmask` was uninitialized, while it was ok in verification mode, hence the stencil verifying but not prob testing.
* Substituting the two fields for their updated counterparts fixed the issue reliably, even with artificially low `cfl_w_limit`
* PR [here](https://github.com/C2SM/icon-exclaim/pull/132).