# Cartesian halo exchanges ###### tags: `cycle 2` ## Assigned to Till (half a cycle) ## Summary For scalability beyond a single node/device capabilities to exchange halo lines are required. This project aims at making a first step into that direction by providing the necessary tooling in Python based on GHex, as well as a prototype implementation for an advection solver. ## Goals __Core__ - Develop basic Python bindings to GHex (MPI transport layer) - Use the Python bindings to develop a prototype, based on MPDATA or parts of it, for simple advection test cases (CPU storages) - Automatic halo exchanges, i.e. without manually initiating them after a stencil call using new storages API __Nice-to-have Goals__ - Support & testing of other transport layers - Support for GPU storages - Understand how and where RDMA can be setup - Understand if sufficient for Elliptic solvers - Open source example independent of MPData __Non-Goals__ - (Automatic) overcomputing based on extent analysis - Incorparation into the GTC IRs - Actual support for RDMA ## Appetite Time: - Full cycle - ~ 50% workforce Resources: - Till - expertise from GHex Team - exchange with Linus with regard to storages API ## Solution __Ghex python bindings__ (40%) Development of a new package wrapping necessary GHex facilities for halo exchanges on Cartesian grids. The package should be independent of GT4Py. _GHex objects to be wrapped_: - Communication object (construction, exchange call) - (Built-in) halo generator - Field wrappers (ideally directly with the new storages API) - Pattern objects __DistributedStorage + stencil decorator__ (20%) The exchange of halo lines can be automated using the new storages API and a custom stencil decorator. The new API includes the definition of halo lines, which could be exchanged prior to a stencil call. Note: Feedback from Vulcan during the design desirable. __Prototype advection solver__ (40%) Implementation of a prototype advection solver based on MPData. The setup of the halo lines is done __manually__ in user code similar to the following snippet: ```python halo_spec = (2, 2, 0) # two symmetric halo lines in i,j, no halos in k # define halo lines for psi field ijk_index_space.define_subdomain(("halo", psi), ijk_index_space["interior"].extend(*halo_spec)) ``` Note: As long as the GT4Py version used by `DistributedStorage` is compatible with MPData the choice of a GT4Py backend does not matter.