# [Blueline] Py2F continuation: Dycore verification, CI/CD on alps/balfrin, muli-process runs, update ICON

Continuation of these: https://hackmd.io/tAfI1iCnQya02KKLmL0VGA,
https://hackmd.io/GIk9YSiUS3C7Cn0XOoJMKw
<!-- Add the tag for the current cycle number in the top bar -->

- Shaped by: Christoph, Sam, Daniel
- Appetite (FTEs, weeks): 1 cycle
- Developers: Samuel, Daniel

## Problem

Currently missing:
* Dycore granule does not yet run in ICON, and is not yet verified.
* CSCS CI for granule build on balfrin/santis utilizing uenv is missing
* Parallel runs are not yet supported by the granules.
* Documentation

## Appetite
1 cycle

## Solution
<!-- The core elements we came up with, presented in a form that’s easy for people to immediately understand -->

#### Dycore granule
- Be able to run the dycore granule on Balfrin (`mch_ch_r04b09`) on CPU and GPU.

- Verify and probtest the dycore granule (GPU, CPU).

- Ensure we can execute experiment with both granules integrated at the same time.

#### CSCS CI on Balfrin/Santis
- Update to newest icon-nwp

- Change `create_target_header` to auto-generated all runscript configurations which are currently added by hand

- CSCS CI on balfrin/santis
- verification mode (compare granule at every execution to Fortran)
- substitution mode (only execute icon4py granule, probtest)
- Switch from mch_ch_r04b09 to small production experiments of MCH (icon1, icon2, kenda1)

#### Parallel Runs

1. We need to modify the diffusion granule wrapper so that it supports running in parallel using GHEX and MPI. There is some preliminary work done already by Abishek in this PR: https://github.com/C2SM/icon4py/pull/457

2. There should be datatests in place on the icon4py side which are able to test parallel runs.

3. Parallel diffusion runs should be verified on the ICON/Fortan side on both GPU and CPU, by passing probtests. For already existing integration work in ICON-EXCLAIM, see: https://github.com/C2SM/icon-exclaim/pull/275

Hints:
- Start with using a LAM experiment that has shown to verify in single core mode such as mch_r04b09_dsl. Then if there is time try to repeat verification first in single core with a new experiment such as APE, and then try the parallel run with the APE experiment.

#### ITIR -> dace
- Test the (ITIR -> dace) backend instead of (ITIR -> gtfn)

## Rabbit holes
<!-- Details about the solution worth calling out to avoid problems -->
Do not try to replicate any workflow and data processing platform. Keep it simple.

## No-gos

No performance optimizations yet.

## Future cycles
- Overhead optimisation
- ...

## Progress
* [x] Parallel runs for diffusion now run and probtest
* [x] Extensive documentation added by Samuel
* [x] Dycore granule now runs in icon-exclaim, also in parallel
* [ ] Dycore verifies and probtests
* [ ] CI/CD for granule integrated blue line