# Continue implementation of embedded Field-View (copy from cycle 17)

- Shaped by: @egparedes, @havogt
- Appetite (FTEs, weeks): 2 devs, full cycle
- Required skills: knowledge of the GT4Py Field-view model and the embedded field-view implementation
- Developers:

## Problem

This is a continuation of the [Embedded Field View](https://hackmd.io/D2gZEUINQ3SVaylITZqEQA) project from cycle 16. The foundations of the embedded field-view have been developed and merged into the `main` branch during cycle 16. However, only a subset of the `gt4py.next` functionality is currently supported, being `remap` the most significant omission.

For the ICON developer workshop in November we will focus on the pieces

## Appetite

The appetite for this project is large because the embedded field-view implementation has to be complete and robust at the end of the cycle, since it will be used in the DWD workshop in November.

## Solution

This project will continue with the implementation of the [design sketched in the previous project](https://hackmd.io/D2gZEUINQ3SVaylITZqEQA#Solution) and partially implemented in Enrique's Embedded Field-View prototype ([EEFV](https://github.com/egparedes/gt4py_embedded/tree/gtnextv2-dispatch)).

### On-going tasks
Features fully or partially implemented with open PRs:

~~- High-level field allocation API (started by @havogt): https://github.com/GridTools/gt4py/pull/1319~~
- Add `where` to embedded field view (started by @havogt): https://github.com/GridTools/gt4py/pull/1316

### Implementation of missing features (in order of priority)
- `remap()`: implementation of unstructured and Cartesian shifts in embedded field-view.
- Part of the work has been already tackled in the [WIP tasks](https://hackmd.io/q2QnqPdSRlqgapGfIyY0tw#WIP-Tasks).
- The algorithm for the unstructured case with NdArray based fields already exists in the [EEFV](https://github.com/egparedes/gt4py_embedded/blob/1f157e3131913dd372198d9901d38d2cef7a945b/src/gtnext/fields.py#L152) prototype.
- The WIP implementation of cartesian shifts is in [PR#1309](https://github.com/GridTools/gt4py/pull/1309)
- `scan()`: Implementation of the current functionality in embedded field-view, which requires using native python `for`-loops.
- Complete the field slicing implementation with a proper high-level API.
- For domain slices, use with the syntax sketched in the [GT4Py team workshop](https://hackmd.io/tE9eaKoOTL-apBRLtEUlJg): `f_v[I(-1):I(5)]`. The more compact alternative `f_v[I[-1:5]]` was discarded (for now) because it can also be interpreted as remapping a field.
- Change the current behavior which returns a scalar value when indexing a single point of the domain, to a [Array API](https://data-apis.org/array-api/latest/)-like scheme returning 0d-arrays. If needed, we could define an extra API point to extract the scalar value of a 0d array (e.g. [`.item()`](https://devdocs.io/numpy~1.23/reference/generated/numpy.ndarray.item)).
- `concat()`: implementation of the new field-view function dealing with boundary conditions by using field concatenation. The implementation of this feature in this project will follow the same restrictions shaped previous cycle. (Check here: [1](https://hackmd.io/D2gZEUINQ3SVaylITZqEQA#concat) and [2](https://hackmd.io/D2gZEUINQ3SVaylITZqEQA#Implement-non-overlapping-concat))

### Fixes and refactorings
- Change `Field[[A,B,C], DType]` typing annotations to a syntactically valid form (`Field[Dims[A,B,C], DType]`) to avoid many mypy complaints in our code, and also as preparation for using variadic generics from [PEP 646](https://peps.python.org/pep-0646/).
- Current annotation style should be still supported to avoid breaking user codes (it should be marked as deprecated at some point).
- For now, `Dims` could be just an alias for `Tuple`.
- FOAST and PAST parsers need to be updated to support the new annotations.
- Probably `Dimension` implementation should be changed to be actual classes to avoid mypy errors and even get some meaningful typing errors for free.
- Add missing `gamma` builtin function (which should be trivial using `FieldBuiltinFuncRegistry`).
- Fix DTypes hierarchy to return dtypes of the proper subclass when using the generic constructor in the superclass. Basically `assert isintance(DType(int32), Int32DType)`)
- Fix typing issues in the Typeguards and the synthetic types used for typeguards

## Rabbit holes

### `concat`

Same rabbit hole as in previous project: implementation of `concat` lowering might be complicated (see [here](https://hackmd.io/D2gZEUINQ3SVaylITZqEQA#concat))

## No-gos

- Full implementation of `concat`

## Progress

- [x] Merge WIP PRs
- [x] Add `where` to embedded field view (PR: https://github.com/GridTools/gt4py/pull/1316)
- [ ] Make `None` backend work in ffront_tests
- [x] basic functionality (https://github.com/GridTools/gt4py/pull/1361)
- [ ] Test all builtin functions with scalars
- [ ] enable test_math_builtin_execution.py
- [ ] Add full set of comparison functions to Field and add tests
- [ ] Add cupy to backend list (Hannes)
- [ ] Add jax to backend list (Hannes)
- [ ] remap + related utils (Enrique)
- [x] Cartesian remap
- [x] unstructured remap
- [x] reductions
- [ ] skip values
- [ ] scan (Hannes)
- [x] implementation
- [x] merge with context var
- [ ] reviewed and merged
- [ ] as_offset (Nikki)
- [ ] function field
- [ ] try nd_array_like function implementation
- [ ] cleanups:
- [ ] implement NamedRange as namedtuple and cleanup element access
- [ ] merge workshop material
- [ ] update examples with a gt4py.next version
- [ ] Implementing Standard Array API for slicing
- [ ] `item()`?
- [ ] Fix issues in dtype hierarchy
- [ ] return concrete subclasses using the generic constructor
- [ ] fix typing issues
- [ ] field slicing: update to new syntax discussed at the workshop
- [ ] concat + at[].set()