# [Greenline] Domain decomposition <!-- Add the tag for the current cycle number in the top bar --> - Shaped by: Magdalena - Appetite (FTEs, weeks): - Developers: <!-- Filled in at the betting table unless someone is specifically required here --> ## Problem <!-- The raw idea, a use case, or something we’ve seen that motivates us to work on this --> There is a utility in `icon4py` `grid_manager.py` that constructs the reads the topology of the ICON grid from a the grid file and constructs further connectivities needed in the model (dycore, diffusion). It only works for single node runs. For parallel runs we depend on reading in the same data from a parallel ICON run on the same grid and the same number of nodes. We want to add functionality for decomposing the grid independently. The exact decomposition algorithm is out of scope for this project. Starting from a decomposition of the cells, which is a mapping of cells to a node number ```latex= d(cell) = node_number ``` we want to distribute the other dimensions (`edge`, `vertex`) and construct the halo regions ## Appetite <!-- Explain how much time we want to spend and how that constrains the solution --> Full cycle ## Solution <!-- The core elements we came up with, presented in a form that’s easy for people to immediately understand --> The original cell distribution can be done by [metis](#metis) or the same format is available as an ascii file from ICON. The implementation should allow to exchange the decomposition algorithm used. - construct halo regions (cell, vertex, edge) - decomposition info: mapping of local to global index, classification of each element of a local field whether it is `owned`, `first` or `2nd` halo line (see `mo_decomposition_tools.f90`) - decompose/distribute fields ### Testing A suitable testing strategy needs to be developed. The most promising approach is through gathering and comparing to global (single node) results ### Draft Exploratory work has been done using the ICON grid using the [xugrid](#xugrid) library which uses [pymetis](#metis) under the hood. See [this notebook](https://github.com/halungge/notebooks/tree/main/ugrid) Apart from the it should be explored whether [atlas](#atlas) provides an easier (ready to be used ) solution to the problem. Also further know how is available from the `gridman` Fortran library developed for ICON-C/Bridge project. ## Rabbit holes <!-- Details about the solution worth calling out to avoid problems --> ## No-gos <!-- Anything specifically excluded from the concept: functionality or use cases we intentionally aren’t covering to fit the ## appetite or make the problem tractable --> For a first version we will concentrate on the global grid, which leaves the extra complexity of incorporating physical boundaries (more flavorful `refin_ctrl` and ordering of values and especially halo values on lateral boundary) for later. ## References #### [metis](https://pypi.org/project/PyMetis/) #### [xugrid](https://github.com/Deltares/xugrid) #### [atlas doc](https://arxiv.org/pdf/1908.06091#figure.caption.4) #### [atlas repo](https://github.com/ecmwf/atlas/tree/develop) ## Progress <!-- Don't fill during shaping. This area is for collecting TODOs during building. As first task during building add a preliminary list of coarse-grained tasks for the project and refine them with finer-grained items when it makes sense as you work on them. --> - [x] Construction of halos: [branch](https://github.com/C2SM/icon4py/tree/halo_construction) - [x] test: simple distribute a field and gather test - [x] direct call of pymetis (without xugrid) - [ ] refactor grid_manager to return local grid - [ ] test: compute stencil with connectivity offset - gather - [ ] geofac_rot, - [ ] geofac_div, - [ ] geofac_n2s - [ ] start/end indices for IconGrid to allow for domain bounds