# Grid Manager
###### tags: `architecture`, `concepts`

## Content
describes a grid manager as it is needed in the (EXCLAIM greenline) python port.
- responsibility of grid manager
- interfaces
- findings, specialities of how ICON handles the grid

## responsibility of grid manager
It should provide information about *where* and *how* the discretized grid are stored, that is
- connectivities for accessing neighboring edges, cells, vertices.
- start-, endindices of different horizontal zones in the grid
- in the case of nesting: access from parent-to-child or child-to_parent in nested grids where applicable.
- decomposed domain: provide information on where the halo regions

### exclusion
- GridManager does not handel vertical information, like number of k-levels.
- GridManager does *not provide* fields containing physical or geometry information about the discretization: that is fields like `inv_vert_vert_length`, lat, lon, cartesian coordinates, cell areas etc. are *not* provided by the GridManager. That is in contrast to the current `t_patch` derived type that provides all of this. These geometrical fields should be kept in semantically more meaningful constructs like `[Edge|Cell|Vertex]-InformationManager` or `-GeometryManager` or made available through the `dataDictionary` or `memoryManager`.
- GridManager does not provide access to MPI Communicators used with a decomposed grid, it only provides information about where the halo regions start and end.

The idea is essentially that:

given any field in ICON the GridManager provides the information necessary to access any (meaningful) subset of that field and calculate any stencils involving this field, no matter of what the physical meaning of the field is and what data it contains.

## Interface

### connectivities (horizontal):
grid manager provides connectivity fields for shifting between horizontal dimensions. For example:

- `gm.get_C2E_connectivity(patch_id)` -> `p_patch(patch_id)%cells%edge_idx(:,:,:)`
- `gm.get_E2C_connectivity(patch_id)` -> `p_patch(jg)%edges%cell_idx(:,:,:)`
- `gm.get_C2E2C_connectivity(patch_id)` -> `p_patch(jg)%cells%neighbor_idx`
- `gm.get_E2C2E_connectivity(patch_id)` -> `p_patch(jg)%edges%quad_idx`
- `gm.get_E2C2V_connectivity(patch_id)` -> `p_patch(jg)%edges%vertex_idx`
* Note: the naming of these methods are up to discussion, these e2c type terminology is base on namings used in icon4py *

### horizontal zone boundaries
GridManager should provide start and end indices for different zones of the horizontal grid, that is essentially the `get_indices_[e|v|c]` based on the `refin_ctl` values.

There is a python implementation of the refin_ctl values, see [HorizontalMarkerIndex](https://github.com/C2SM/icon4py/blob/greenline-dycore/atm_dyn_iconam/src/icon4py/diffusion/horizontal.py) (https://github.com/C2SM/icon4py/blob/greenline-dycore/atm_dyn_iconam/src/icon4py/diffusion/icon_grid.py)
The naming of the regions there are based on some work that was done by MCH for icon-exclaim and dusk/dawn.

def get_end_index(self, dim: Dimension, marker: HorizontalMarkerIndex) -> int:
    """Return end index of the region described by the 'marker' in a horizontal (local) field.
    
    This index can be used to specify the domains of stencil calculations in in gt4py. 
    
    """
    pass
def get_start_index(self, dim: Dimension, marker: HorizontalMarkerIndex) -> int:
    """Return start index of the region described by the 'marker' in a horizontal (local) field.
    
    This index can be used to specify the domains of stencil calculations in in gt4py. 
    """
    pass
 

### improvement of the interface
- (Tuples) in python the boundary domains are called upfront and then passed to `GT4Py program`s, it would be simpler to pass a tuple of `Markers` and return a tuple of start- or end indices
- find a way to encapsulate the `HorizontalMarkerIndex` and trade it for better readability.

### differences to ICON (Fortran)

The python implementation has no blocking in the Frontend, for `GT4Py` we need the unblocked indices, that is indices into 1 dimensional horizontal arrays which corresponds to 1 single block in the Fortran code.

NOTE: @Magdalena: need to think more about that! especially the double loop structure in Fortran makes it complicated

NOTE: Also depending on the implementation the 0 vs 1 based of indices should be taken into account.

call get_local_boundary_blocks_edges(startblk_lb, endblk_lb)
call get_nudging_blocks_edges(startblk_ndg, endblk_ndg)
do jb = startblk_lb, endblk_ndg
	call get_local_boundary_indices_edges(jb, startblk_lb, endblk_ndg, startidx, endidx)
	do idx = startidx, endidx

## Questions/Discussion:
- Does the grid manager read the grid file(s) directly? -> yes.
- Vertical grid Information: should we completely separate those two?
- (@Magdalena) understand the grid (re)distribution of the grid (among PE)?
- Does the grid file contain the entire grid or is there one file per PE? -> the grid file contains the entire grid.
- Is there a IO node that does all the reading and then redistributes?

## ICON grid files
- NetCDF format
- each file contains information of grid (that is a grid with a certain resolution). Nested grids are stored in separate files and contain information on the relation to their parent grid (see below).
### data in grid files (non-exhaustive list)

| Field name | type/dimension | description | managed by grid manager? (y/n) |
| ----------------------------------------------------------- |:-------------------- | --------------------------------------------------------------------------------------------------------------- | ------------------------------ |
| `cell` | `integer` | number of cells | y |
| `edge` | `integer` | number of edges | y |
| `vertex` | `integer` | number of vertices | y |
| `nc` | `integer` | number of vertices neighoring an edge,i.e. 2 | |
| `nv` | `integer` | number of edges/vertices neighoring a cell,i.e. 3 | |
| `no` | `integer` | number of neighoring cells on a cell (including origin),i.e. 4 | |
| `ne` | `integer` | number of vertices/edges/cells neighoring a vertex (for hexagon),i.e. 6 | |
| `clon`, `clat` | `double[cell]` | lon, lat of the midpoints of the cells | n |
| `elon`, `elat` | `double[edge]` | lon, lat of the midpoints of the edges | n |
| `vlon`, `vlat` | `double[vertex]` | lon, lat of the vertices | n |
| `cell_area` | `double[cell]` | triangle areas | n |
| `clon_vertices`, `clat_vertices` | `double[3, cell]` | | |
| `meridional_normal_primal_edge`, `zonal_normal_primal_edge` | `double[edge]` | components of at the edge midpoint | n |
| `vertex_of_cell` | `integer[3,cell]` | the indices neighboring vertices to a cell: `vertex_of_cell(:,i)` are the 3 vertices adjacent to cell `i`:`c2v` | y |
| `cells_of_vertex` | `integer[3, cell]` | neighbouring edges to a cell:`v2c` | y |
| `edge_of_cell` | `integer[3, cell]` | neighbouring edges to a cell:`c2e` | y |
| `adjacent_cell_of_edge` | `integer[2, edge]` | neighboring cells to an edge: `e2c`, | y |
| `edge_vertices` | `integer[2, edge]` | neighboring vertices to an edge: `e2v` | y |
| `cells_fo_vertex` | `integer[6, vertex]` | indices of neighboring cells to a vertex:`v2c` | y |
| `edges_of_vertex` | `integer[6, vertex]` | indices of neighboring edges to a vertex: `v2e` | y |
| `neighbor_cell_index` | `integer[3, cell]` | neighboring cells to a cell: `c2e2c` | y |
| `vertices_of_vertex` | `integer[6, vertex]` | indices of neighboring vertices to a vertex along edges: `v2e2v` | y |
| `zonal_normal_dual_edge`, `meridional_normal_dual_edge` | `double[edge]` | components of the normal vector at the facets of the dual control volume | n |
| `uuidOfHGrid` | `uuid` | id of the grid | y |

#### nesting specific
Nesting means there is a further subdivision by 2 (of an area of) the grid: each parent edge is split into 2, which splits each triangle cell into 4 triangles
*child-to-parent* relationship is stored in the grid file, *parent-to-child* is computed.

| Field name |type/dimension | description | managed by grid manager? (y/n) |
|---------------|------------|---------------------------------------|---|
|`uuidOfParHGrid`|`uuid`| id fo the parent grid, can alternatively be read from the namelist(deprecated): `grid_nml:dynamics_parent_grid_id`|y|
|`parent_cell_index`| `integer[cell]`| global index of the cell in the parent grid, where this cell is part of: `parent_cell_index(i)` returns an index into any cell field of the grid with id `uuidOfParHGrid`|y|
|`parent_edge_index`|`integer[edge]` | global index of the edge in the parent grid, where this edge is part of. |y|
|`parent_vertex_index`| `integer[vertex]`| global index of parent vertex in the parent grid. ?? what does this actually mean?? |y|

## ICON `t_patch`
*TODO: add a list of fields that should go into the gridmanager and whether they are read from grid file or calculated and where*

### Connectivity FIelds: ICON gridfile vs `t_patch`

In some cases ICON applies post processing to connectivity fields. This has been checked by comparing fields read from the gridfile with the corresponding connectivity fields serilialized from `t_patch`. These differences are documented here for the `mch_ch_r04b09_dsl`experiment (single node, limited area model!)

#### INVALID grid points
For limited area and global models there are irregular points at the boundary (limited area) or at the pentagon points in the global grid, where a vertex has only 5 and not 6 neighboring cells and edges. In some cases ICON substitutes those with the index of the last valid neighbor, which occurs then twice in the neighbor list. When used in neighbor sums it is the responsiblity of the user to ensure that coefficient fields are set to zero.

| Icon4Py name | gridfile name | invalid values in grid file | invalid values in ICON | description |
| ------------ |:--------------------- |:--------------------------- |:---------------------- |:------------------------------------------------------------------------------------------------------------- |
| `V2C` | cells_of_vertex | y | n | pentagon points, or boundary vertices do not have 6 neighbouring cells. ICON uses "last valid neighbor" trick |
| `V2E` | edges_of_vertex | y | n | pentagon points, or boundary vertices do not have 6 adjacent edges. ICON uses "last valid neighbor" trick. |
| `E2V` | edge_vertices | n | n | no loose edges exist in the domain. |
| `E2C` | adjacent_cell_of_edge | y | y | limited area: boundary edges have only one adjacent cell. ICON keeps those as invalid indices. |
| `C2E` | edge_of_cell | n | n | each cell in a domain has exactly 3 adjacent vertices |
| `C2V` | vertex_of_cell | ? | ? | |

#### other differences
The grid manager constructs the field for the diamond vertices `e2c2v` ICON adds the far vertices only for points in the interior of the grid, for the boundary edges there are only the two adjacent vertices defined. The far edge on the existing cell is missing.