# Continue implementation of embedded Field-View

- Shaped by: @egparedes, @havogt
- Appetite (FTEs, weeks): 2 devs, full cycle
- Required skills: knowledge of the GT4Py Field-view model and the embedded field-view implementation
- Developers:

## Problem

This is a continuation of the [Embedded Field View](https://hackmd.io/D2gZEUINQ3SVaylITZqEQA) project from last cycle. The foundations of the embedded field-view have been developed and merged into the `main` branch during last cycle. However, only a subset of the `gt4py.next` functionality is currently supported, being `remap` the most significant omission.

Furthermore, there is an important workshop planned at DWD for the end of November where embedded field-view execution is required for letting the users successfully engage with gt4py.

## Appetite

The appetite for this project is large because embedded the field-view implementation has to be complete and robust at the end of the cycle, since it will be used in the DWD workshop in November.

## Solution

This project will continue with the implementation of the [design sketched in the previous project](https://hackmd.io/D2gZEUINQ3SVaylITZqEQA#Solution) and partially implemented in Enrique's Embedded Field-View prototype ([EEFV](https://github.com/egparedes/gt4py_embedded/tree/gtnextv2-dispatch)).

### On-going tasks
Features fully or partially implemented with open PRs:

- Update embedded Iterator implementation to use the new embedded Field implementation (started by @havogt): https://github.com/GridTools/gt4py/pull/1308
- High-level field allocation API (started by @havogt): https://github.com/GridTools/gt4py/pull/1319
- Add `where` to embedded field view (started by @havogt): https://github.com/GridTools/gt4py/pull/1316

### Implementation of missing features
- `remap()`: implementation of unstructured and cartesian shifts in embedded field-view.
- Part of the work has been already tackled in the [WIP tasks](https://hackmd.io/q2QnqPdSRlqgapGfIyY0tw#WIP-Tasks).
- The algorithm for the unstructured case with NdArray based fields already exists in the [EEFV](https://github.com/egparedes/gt4py_embedded/blob/1f157e3131913dd372198d9901d38d2cef7a945b/src/gtnext/fields.py#L152) prototype.
- The WIP implementation of cartesian shifts is in [PR#1309](https://github.com/GridTools/gt4py/pull/1309)
- `concat()`: implementation of the new field-view function dealing with boundary conditions by using field concatenation. The implementation of this feature in this project will follow the same restrictions shaped previous cycle. (Check here: [1](https://hackmd.io/D2gZEUINQ3SVaylITZqEQA#concat) and [2](https://hackmd.io/D2gZEUINQ3SVaylITZqEQA#Implement-non-overlapping-concat))
- `scan()`: Implementation of the current functionality in embedded field-view, which requires using native python `for`-loops.
- Complete the field slicing implementation with a proper high-level API.
- For domain slices, use with the syntax sketched in the [GT4Py team workshop](https://hackmd.io/tE9eaKoOTL-apBRLtEUlJg): `f_v[I(-1):I(5)]`. The more compact alternative `f_v[I[-1:5]]` was discarded (for now) because it can also be interpreted as remapping a field.
- Change the current behavior which returns a scalar value when indexing a single point of the domain, to a [Array API](https://data-apis.org/array-api/latest/)-like scheme returning 0d-arrays. If needed, we could define an extra API point to extract the scalar value of a 0d array (e.g. [`.item()`](https://devdocs.io/numpy~1.23/reference/generated/numpy.ndarray.item)).

### Fixes and refactorings
- Change `Field[[A,B,C], DType]` typing annotations to a syntactically valid form (`Field[Dims[A,B,C], DType]`) to avoid many mypy complaints in our code, and also as preparation for using variadic generics from [PEP 646](https://peps.python.org/pep-0646/).
- Current annotation style should be still supported to avoid breaking user codes (it should be marked as deprecated at some point).
- For now, `Dims` could be just an alias for `Tuple`.
- FOAST and PAST parsers need to be updated to support the new annotations.
- Probably `Dimension` implementation should be changed to be actual classes to avoid mypy errors and even get some meaningful typing errors for free.
- Add missing `gamma` builtin function (which should be trivial using `FieldBuiltinFuncRegistry`).
- Fix DTypes hierarchy to return dtypes of the proper subclass when using the generic constructor in the superclass. Basically `assert isintance(DType(int32), Int32DType)`)
- Fix typing issues in the Typeguards and the synthetic types used for typeguards

## Rabbit holes

### `concat`

Same rabbit hole as in previous project: implementation of `concat` lowering might be complicated (see [here](https://hackmd.io/D2gZEUINQ3SVaylITZqEQA#concat))

## No-gos

- Full implementation of `concat`

## Progress

- [ ] Merge WIP PRs
- [x] embedded Iterator implementation using new field (PR: https://github.com/GridTools/gt4py/pull/1308)
- [ ] Add `where` to embedded field view (PR: https://github.com/GridTools/gt4py/pull/1316)

- [ ] Redesign field allocation API for next (PR: https://github.com/GridTools/gt4py/pull/1319)
- [ ] Create new `DataField` (Domain + DLPack buffer) which gets returned when allocating fields with `.empty` or `asfield`

       class GTDomainField(Protocol):

            @property
            def __gt_domain__(self) -> Domain:
                ...
            
            # dlpack methods

Also extend `cartesian` to support the new interface as a fallback case
- [ ] `np_as_located_field` should return the new `DataField` temporarily
- [ ] Make `DataField` compatible with the current next bindings
- [ ] Wrap `DataField` inside `EmbeddedNDArrayField` for `iterator.embedded` execution in the entry point
- [ ] Delete `np_as_located_field`

- [ ] Fix issues in dtype hierarchy
- [ ] return concrete subclasses using the generic constructor
- [ ] fix typing issues
- [ ] field slicing
- [ ] remap
- [ ] concat
- [ ] scan