# Visit DWD Offenbach 11.5.-12.5.

## 11.5. Morning: Icon Grid
- EXCLAIM: Magdalena Luz, Will Sawyer
- DWD: Florian Prill

We went through Florians [bridge code](https://gitlab.dkrz.de/fprill/code-snippets) in order to understand the way he sets up the grid and does domain decomposition.

### notes

- the grid generator is in [ICON tools](https://gitlab.dkrz.de/dwd-sw/dwd_icon_tools).
- **spring dynamics** is done in the grid generator.
- **cell areas**: Icon uses something between flat (triangle area) and spherical. In Bridge code by Florian there are all three options. (flat, spherical, icon)
- The grid file contains many fields that can be derived from a minimal subset, (that is done in the [Bridge code](https://gitlab.dkrz.de/fprill/code-snippets) that Florian uses for DG.)

#### grid properties
- numbering of cells for nested subgrids: (see `mo_model_domain.f90`)

! -----------------------------------------------------------------------------
  ! grid_cell class - corresponds to triangles
  ! -----------------------------------------------------------------------------
  !
  ! This data type holds the topological and geometrical information on
  ! the grid cells, read from the NetCDF grid file.
  !
  ! * Note on parent-child relation "child_idx/child_blk":
  !
  !          *---------------o--------------*
  !           \__   4     _/  \_    2    __/
  !              \_     _/   3  \_    __/
  !                \__ /__________\ _/
  !                   o            o  
  !                     \__  1  __/
  !                        \_ _/
  !                          *
  !
  !   The ICON model makes an important implicit assumption on the
  !   child ordering:
  !
  !   - cell child #3 is the "interior" child (containing the parent
  !     cell's mass point)

- boundaries: in the *ordered* part of the grid (lateral boundary) there are 14 cell rows (`max_refin_c_ctrl = 14`) why this number was chosen is unclear. So the `refin_c_ctl` values determines the distance from the boundary.
- halo cells are added at the end of the fields except for those that are part of the lateral boundary the are contiguous with the other later boundary cells. But not even Günther knows how many of them exactly, *lateral boundary cell rows 1 and 2 tend to be in the boundary region, rows 3 to ... in the halo region*
- vertices around a cell (vertex-of-cell array in grid file): indices are ordered counter clockwise.
- rhs: `(<t>, <n>, <k>) ` is a right handed system, where
- `<t>` tangent vector `edge_vertices(2) - edge_vertices(1)`
- `<n>` primal nomral vector
- `<k>` outward pointing vector of the tangential plane

#### domain decomposition
- Bridge code for DG builds up a kd-tree.
- Algorithm in ICON: very old algo, probably nobody (?) who really understands it out of the box
- **ICON** - `mo_setup_subdivison.f90`: processor splitting (user determines which patch goes to which compute node) should be removed.
- There are 2 halo cell rows.

## 11.5. Morning: Tracer advection
- EXCLAIM: Andreas Jocksch
- DWD: Daniel Reinert

## 11.5. Afternoon: Dycore session 1
- EXCLAIM: Will Saywer, Abishek Gopal, Christoph Mueller, Daniel Hupp, Andreas Jocksch, Magdalena Luz
- DWD: Florian Prill, Guenther Zaengl, Daniel Reinert, Marek, Christoph (?)

We essentially went through solve `mo_solve_nonhydro.f90` and `mo_velocity_advetion.f90` on the basis of [our notes](https://hackmd.io/1qQS741WRom4l-lskqm8Yw) that we had prepared before and addressed all the questions noted there. Answers are written down there or put into the [glossary](https://hackmd.io/PALNc5ecTv6Gz-md5tipNw).
We also identified some parts of the code that could be refactored or removed in ICON itself, we leave this task to DWD developers though. See [our notes](https://hackmd.io/1qQS741WRom4l-lskqm8Yw).

### TODOs resulting from this session

* For production MCH should log if critical CFL of 0.85 and 1.05 are exceeded, for EXCLAIM, clearify if we need to care about this
* Confirm if thinly populated masks actually bad for NEC or new compiler version magically solves this issue as some claim it does.
* Request shaping of project to introduce nproma_c, nproma_e and nproma_v?
* Florian: Meshing library should be rather simple and should not have very strict post conditions so you can substitute it and the code still works
- This clashes with our view that if the meshing has strong post conditions, we can do stronger optimizations
- Would have to proof the potential of stricter meshing tool with toy example as first step

## 12.5. Morning: Presentation gt4py/ icon4py
- see above
- remote: Luis Kornblueh, Claudia Frauen, Ralph, possibly more from DKRZ, MPI, Nikki (Exclaim)

Short summary: presentation of `gt4py`, blue line (`liskov`) and green line. Slides are [here](https://docs.google.com/presentation/d/1q_U6f0yk9aWyV6SsVrCq9XmqCCs_DvAsEe_3x1JTdq8/edit#slide=id.g23a88cde77a_0_165).

### general impression (Magdalena)
- There were lots of questions and discussion on `gt4py` many of them anicipating what we (Christoph) was going to explain next. I think this showed there was really a lot of interest.
- In general, there is a bit of reluctancy of having to learn a new language and an environment they are not yet familiar with.
- They seem to be quite open to the blue line approach and take it over if we reach some speed up. Even suggesting that the original Fortran kernels might be removed if the DSL performs better or have an "inverse-Liskov" from where you can reach the original code if you want to (even Guenther would be okish with that).
- There is lots of skepticism on the greenline. First of all, because they don't want to fully switch environments and greenline will probably for a long time not support all features they need (f.ex. nesting) and still having a Python ICON port would mean to support two models.

### Questions/Feedback during presentation to DWD (Abishek)

1. Gunter: Nesting is really important.

2. All: half dim handling (not sure abt this, please elaborate)

3. Gunter: Backend for NEC is important to DWD

4. Marek: questions about () required for correctly broadcasting dims CellDim * KDim . Scientists may not want to deal with this complexity

5. Question about naming of sparse dims. Most find it counter-intuitive, from a model developer perspective

6. Do we plan to allow for tensor stencils. Was a problem in Dawn.

8. Claudia: Do we need to adopt changes to the backend after adding new/changing stencils. Our response: some features may not be available in all backends, so possibly..

9. How to replace the where clauses in cells with multiple domains. perhaps use markers?

10. Separation of concerns may backfire for more traditional model development centers. How can science budgets afford HPC engineers. Previously scientists also used to optimize at a somewhat higher level, and leave the dirty work to the compiler vendors. With DSLs, that model will change.

11. Questions about green-line diverging from ICON. Our response: Blue-line is intended for scientific use cases for the next few years. Green-line is highly experimental.

12. What about code branches that aren't taken (in both blue and green lines). who will support it in the future?

### 12.5. Afternoon: Dycore session 2
- participants see above

Quick go throught `mo_solve_nonhydro.f90` where we wrote down a (physical) name or discription for each loopnest, from start of the `solve_nh` routine down to the solver.

Christoph Müller has a list of these names. [TODO]