# Architecture for Models with GT4Py [minutes]

-> Slides(password: the slack channel name without `#`): https://polybox.ethz.ch/index.php/s/SZUHMAYJbsWn0mQ

## Thursday 6th June

### 9:30 - 11:00 Warmup session and introductions

- Oliver E.: Lagrangian to Eulerian use-case forced the introduction of the `while` loop in `gt4py.cartesian`
- Florian: probably we need a new `Column` concept to deal with column-based physics code
- Hannes: staggering in cartesian grids should be easy and automatic (e.g. `X + Half = X^ ...`)
- Rusty: managing a team with many people, some of them looking into improving gt4py.cartesian performance by inspecting at the generated source code
- Loic (MeteoFrance): porting AROME physical parametrizations for PMAP in gt4py.cartesian. Some other people trying Loki as a first step into the conversion. Already using python infrastructure like Sympl (ifs-physics-common), xarray. Some challenges with gt4py:
- choice of stencil at compilation (inline one or other computation)
- choice of function behavior at run-time
- `while` loops for explicit sources (field collections?)
- `while` loops for explicit sources computation (GlobalTables)
For the future keep looking at Loki as pre-translation step for GT4Py, and also looking SURFEX.
- Christoph: interested in the interface between Fortran and Python and performance. At MCH they wrote a Fortran pre-processor to substitute loop nests by calls to gt4py stencils in the original Fortran ICON code.
+ Florian comment: sometimes DaCe doesn't fuse loops that should be fused because of some carried state prevents merging them. It should be improved.
- Xavier: interested in how to address the control flow issue in the solver or trying different components. GT4Py should also improve in the user friendliness for writing physics and column-based physics.
- Till: working in gt4py IR and contact person for PMAP.
- Sara: working in PMAP and gt4py. Some issues in the past with prototyping in embedded (can’t run without parsing/using parts of frontend).
- Jeremy: he started working in Sympl and later in Pace. Now doing mostly ML climate models. Interested in models architecture and patterns like the `Builder` pattern:
- read configuration from files and represent it as classes (_probably_ similar to `omegaconf`)
- code sample: https://storage.googleapis.com/vcm-ml-public/jeremym/fme-docs/5b9e2b80/builder.html
- Chia Rui: domain scientist at EXCLAIM. Bad experiences with error messages (in embedded and compiler backends) when something goes wrong. For example, I encountered memory access problem when running on GPU in one stencil and I did not know which line causes the problem.This is especially bad when stencil is long. Also a bit frustated with that gtp4y does not allow while loop iteration as physics parameterization involves looping over species and some iterative schemes.
- Mikael Stellio: started with gt4py as a student long time ago. Ported FV3 microphysics and some other codes to gt4py in different student projects. Currently working in HAMMAN project (porting aerosol model to OpenACC and gt4py).
+ He encountered weird that `scan_operator` signature accepts scalars but then are called with fields.
- Stefano: long-time (since 2016) GT4Py user and used it for this PhD. Currently working in PMAP physical parametrizations.
+ Whishlist:
+ vertical offset writes
+ lower-dim temporaries
+ absolute indexing in the vertical
+ `for` loops along data dimensions
- Christos: joined CSCS in October. Working in DaCe orchestration for gt4py and halo exchanges.
- Magdalena: joined EXCLAIM 2 years ago and started with gt4py by porting stencils. Currently working mostly in the architecture and components of a python model. Annoyed by the ugly way to deal with vertical staggering currently in gt4py and two ways to express sparse fields.
- Enrique: working in gt4py since 2016 and interested in making gt4py models cleaner and better structured.
- Abishek: working in EXCLAIM for 2 years. Integrating the gt4py-python granule in ICON. The current stencils in icon4py do not look very readable.
- Mauro: director of software of EXCLAIM and original author of GridTools. Interested in the composability and architecture of a full model using components. Also interested in being able to define the data layout from the computations that are going to be performed, and in how to place the halo exchanges automatically for better performance.

### 11:15 - 12:30 Current infrastructure & support libraries

#### PMAP infrastructure

- PMAP-L (cartesian):
- IndexSpaces, `stencil` wrappers, `FieldDescr` field descriptors for linking index spaces and field shapes
- partial implementation of generics, permit a number of pinput locations, output location implements as `SameAs` input location
- PMAP-L (distributed):
- Partitioner, HaloEschanger, DistributedField
- Context manager to allocate temporary fields (fields pool)
- Context manager for halo exchanges
- CI:
- steps: build docker image, compilation cache for the compiled stencils, execution
- Tests:
- basic unit testing
- integration tests: several cases, some automation for validation but also manual validation by scientists (Christian K.)
- pytest-benchmarks (no yet place for the outputs)
- Configuration:
- Read from yaml files and create `pydantic` models (omega-conf like?)
- Wishlist:
- Compilation cache
- Field collections
- Enums
- Boundary conditions
- Generics
- Convergence loops with global reductions

#### NDSL / PACE infrastructure
-> Slides: _arch_workshop24_ndsl.pdf_

- SHiELD = GFDL's FV3 (kind of forked from the NOAA's UFS dycore?)
- Pace = PyFV3 + PySHiELD + NDSL
- NDSL middleware layer
- constants, metric terms
- wrapper for gt4py and caching, DaCe support
- MPI interface
- Configuration classes with overlapping sets as properties
- Quantity and Stencil factories
- `Translate_` classes initially used for testing the migration from Fortran but now are also useful for having standalone execution of some components.

#### ICON4Py infrastructure with Python driver (green-line)

- EXCLAIM project uses ICON model from DWD, now also in production at MeteoSwiss.
- ICON4Py repository is a monorepo with namespace packages for different components
- ICON Fortran is not really modular although there is some ongoing efforts. A _"granule"_ is the concept of module for Fortran
- Icon4py granules follow currently the same interface as the Fortran granules. The component object has an explicit `init` method different than `__init__()` because of it.
- Python components can now be called from Fortran as a black box, which would allow changing the internals of the module.
- Currently exploring reading the ICON grid file using UXarray
- GHEX used for halo exchanges.

## Friday 7th June

### **9:15** Jeremy's summary
- Florian: code coverage tools could be useful, specially focused in testing all the numeric components in the testing simulation.

### **10:00** Shared Libraries
- (Jeremy) Suggestions to add to GT4Py.
- Subdomain definitions:
- The PMAP domain definitions using names looks better and not so lower-level as the horizontal regions in FV3
- They might be a good way to combine both ways
- User-provided cache implementations to customize the behavior.
- Try to identify general functionality in the support libraries that it could be extracted and reused.
- Checkpointers from Pace
- Parallel compilation and distribution of stencils/programs

### **11:00** Physics
- Necessary to write full models and therefore for user's adoption of gtp4y.
- Mikael (HAMAM) already collected some patterns and how to migrate them (if possible) to gt4py
- ECMWF have collected/will collect some patterns for radiation? and tried different frameworks/languages

#### Action points:
- Create a repository with examples of column physics patterns. All models to contribute.
- https://github.com/GridTools/physics_patterns
- Match every examples with an answer in gt4py.next and gt4py.cartesian.
- Clean it up to make it digestible for new users of GT4Py (e.g. date the example & their matched implementation, etc.)

### **12:00** Other topics

#### Autodifferentiation
- Florian and someone also at NASA looked into using JAX with the numpy backend but didn't work very well
- Another try they did was using Enzyme (LLVM autodifferentiation) with the generated C++ by the DaCe backend.
- SPCL is also doing some early work in autodifferentiation with SDFGs
-