---
# System prepended metadata

title: Cartesian halo exchanges
tags: [cycle 2]

---

# Cartesian halo exchanges

###### tags: `cycle 2`

## Assigned

to Till (half a cycle)

## Summary

For scalability beyond a single node/device capabilities to exchange halo lines are required. This project aims at making a first step into that direction by providing the necessary tooling in Python based on GHex, as well as a prototype implementation for an advection solver.

## Goals

__Core__

- Develop basic Python bindings to GHex (MPI transport layer)
- Use the Python bindings to develop a prototype, based on MPDATA or parts of it, for simple advection test cases (CPU storages)
- Automatic halo exchanges, i.e. without manually initiating them after a stencil call using new storages API

__Nice-to-have Goals__

- Support & testing of other transport layers
- Support for GPU storages
- Understand how and where RDMA can be setup
- Understand if sufficient for Elliptic solvers
- Open source example independent of MPData

__Non-Goals__

- (Automatic) overcomputing based on extent analysis
- Incorparation into the GTC IRs
- Actual support for RDMA

## Appetite

Time:

- Full cycle
- ~ 50% workforce

Resources:

- Till
- expertise from GHex Team
- exchange with Linus with regard to storages API

## Solution

__Ghex python bindings__ (40%)

Development of a new package wrapping necessary GHex facilities for halo exchanges on Cartesian grids. The package should be independent of GT4Py.

_GHex objects to be wrapped_:

- Communication object (construction, exchange call)
- (Built-in) halo generator
- Field wrappers (ideally directly with the new storages API)
- Pattern objects


__DistributedStorage + stencil decorator__ (20%)

The exchange of halo lines can be automated using the new storages API and a custom stencil decorator. The new API includes the definition of halo lines, which could be exchanged prior to a stencil call.

Note: Feedback from Vulcan during the design desirable.

__Prototype advection solver__ (40%)

Implementation of a prototype advection solver based on MPData. The setup of the halo lines is done __manually__ in user code similar to the following snippet:

```python
halo_spec = (2, 2, 0) # two symmetric halo lines in i,j, no halos in k

# define halo lines for psi field
ijk_index_space.define_subdomain(("halo", psi), 
  ijk_index_space["interior"].extend(*halo_spec))
```

Note: As long as the GT4Py version used by `DistributedStorage` is compatible with MPData the choice of a GT4Py backend does not matter.