# Refactor icon4py repo
###### tags: `cycle 15`
<!-- Remove `template` tag and add `cycle XX` -->
- Shaped by: Sam, Magdalena
- Appetite: 3 weeks, 1 developer
- Developers:
## Problem
The icon4py repository has grown to contain very different tools and code: individual stencils, model driver code, tools for interfacing with icon-exclaim. The original structure with individual subfolder matching the icon structure no longer suits this set up.
## Appetite
3 weeks
## Solution
### Folder/Package Restructuring
Use hierarchical folder structure instead of multiple namespace packages. For example:
- `model`: Contains all model subpackages (e.g. `diffusion`, `advection`, `driver`, `common`)
- `tools`: Contains all other tools to work with icon4py code (e.g. `liskov`, `icon4pygen`, `f2ser`)
#### ICON4Py Model
Use different distribution packages for every granule (inside `icon4py` namespace)
- Folder layout:
```
icon4py/
model/
common/
src/icon4py/model
common/
__init__.py
pyproject.toml # distribution package: icon4py-common
diffusion/
src/icon4py/model
diffusion/
__init__.py
pyproject.toml # distribution package: icon4py-diffusion
advection/
src/icon4py/model/
advection/
__init__.py
pyproject.toml # distribution package: icon4py-advection
pyproject.toml # distribution package: icon4py-model ??
driver/
src/icon4py/
driver/
__init__.py
pyproject.toml # distribution package: icon4py ? icon4py-driver ?
```
- Installation:
```
pip install icon4py ? icon4py-driver ?
pip install icon4py-diffusion # most likely included as dependency in icon4py
pip install icon4py-advection # most likely included as dependency in icon4py
```
- Usage:
```python
import icon4py.model.diffusion
import icon4py.driver
...
```
---
#### ICON4Py Tools (mostly _blue line_)
Single distribution package for all tools
- Folder layout:
```
tools/
src/icon4pytools/
__init__.py
icon4pygen/
__init__.py
liskov/
__init__.py
f2ser/
__init__.py
pyproject.toml # single distribution package: icon4pytools
```
- Installation:
```
pip install icon4pytools
```
- Usage:
```python=
import icon4pytools.icon4pygen
import icon4pytools.liskov
...
```
---
### Main Tasks
#### Sync with Spack package maintainers
Refactoring the structure or renaming packages will break the icon4py spack package. Changes should be discussed and communicated to the spack package maintainers to avoid problems.
#### Unit tests
For each stencil unit test we currently call the corresponding stencil `program`. All stencil unit tests should be refactored to call the corresponding `field_operator` instead.
#### ICON4Py Model
Create a new package `icon4py` which contains all code related to the ICON4Py Model. All stencils for each component of ICON (e.g. `atm_dyn_iconam`) should be stored under `icon4py.model`.
Driver code can be stored in `icon4py.driver`. There should also be an `icon4py.common` subpackage containing the current package.
As a follow-up tasks we could also refactor stencils which do the same computation and make them accessible via a `shared` stencil package, for example:
- Stencils which set a field to zero
#### ICON4Py Tools
Create a new package `icon4pytools` containing all the CLIs for integration of icon4py code into ICON. This will contain `icon_liskov`, `f2ser`, `icon4pygen`, and any other future CLIs or tools developed for the blue line.
Additionally we should standardise the interfaces and help messages of the CLI tools to use a common format/language.
#### Improve test suite
The `testutils` package should be expanded with subpackages for containing helpers/utils for the test suite of each subpackage. For example:
```
testutils/
liskov/
helpers.py
...
f2ser/
fortran_samples.py
...
icon4py/
dimension.py
...
```
#### General Cleanup Work
- Ensure that each installable subpackage has its own README, and that each README is up to date.
- Ensure that the root-level README of the repository is up to date.
- All obsolete or stale branches or PRs should be deleted or closed respectively.
- Complete the about section in the repository.
#### Move to a forking workflow
Currently collaborators checkout branches and push them directly to the repository. We would like to implement a forking model. As the repository is private only members of the team (with the correct permissions) are allowed to fork. Communicate who needs access and communicate that we will use a forking model.
#### MyPy Support
Currently MyPy is not run correctly in the Github Workflows. It also is not run correctly when `precommit` is invoked. MyPy needs to run on all the source code in both cases.
This involves:
- Adding MyPy to the QA pipeline.
- Adding MyPy to precommit correctly so that it runs over all source files for each subpackage. With the current repo structure MyPy can currently be invoked in the following way: `MYPYPATH=pyutils/src mypy --explicit-package-bases pyutils/src/**/*.py
`
This will have to be changed to reflect the new structure.
#### Introduce versioning
Ensure there is a global `__version__` that is exported by each package. Introduce `bumpversion` to bump the versions for each package.
## Rabbit holes
<!-- Details about the solution worth calling out to avoid problems -->
- Create a first test case of the new folder structure before moving all the files to fix problems early if they occur.
- Once MyPy is implemented correctly it is likely that some static typing problems will be flagged up, which could be time consuming to resolve. Thus it is advisable to focus on this part after having covered all the other points.
## No-gos
<!-- Anything specifically excluded from the concept: functionality or use cases we intentionally aren’t covering to fit the ## appetite or make the problem tractable -->
Do not use this project to refactor or improve the source code.
---
---
## Other options for Model:
### Single distribution package for all granules
- Folder layout:
```
icon4py/
src/icon4py/
__init__.py
model/
__init__.py
difussion/ # it contains mostly gt4py stencils
__init__.py
stencils.py
...
driver.py # driver/ ...
pyproject.toml # single distribution package: icon4py
```
- Installation:
```
pip install icon4py
```
- Usage:
```python=
import icon4py.model.diffusion
import icon4py.driver
...
```
---
---
## Other Options for Tools:
### Different distribution packages for every single tool (inside `icon4pytools` namespace)
- Folder layout:
```
tools/
icon4pygen/
src/icon4pytools/
icon4pygen/
__init__.py
pyproject.toml # distribution package: icon4pytools-icon4pygen
liskov/
src/icon4pytools/
liskov/
__init__.py
pyproject.toml # distribution package: icon4pytools-liskov
f2ser/
src/icon4pytools/
f2ser/
__init__.py
pyproject.toml # distribution package: icon4pytools-f2ser
```
- Installation:
```
pip install icon4pytools-icon4pygen
pip install icon4pytools-liskov
pip install icon4pytools-f2ser
```
- Usage:
```python=
import icon4pytools.icon4pygen
import icon4pytools.liskov
...
```
### Different distribution packages for every single tool (no common namespace)
- Folder layout:
```
tools/
icon4pygen/
src/icon4pygen/
__init__.py
pyproject.toml # distribution package: icon4pygen
liskov/
src/liskov/
__init__.py
pyproject.toml # distribution package: liskov
f2ser/
src/f2ser/
__init__.py
pyproject.toml # distribution package: f2ser
```
- Installation:
```
pip install icon4pygen
pip install liskov
pip install f2ser
```
- Usage:
```python=
import icon4pygen
import liskov
...
```
-----
# TODO
- [x] **Sync with Spack package maintainers**
- [x] Discuss and communicate any changes in package structure or renaming with spack package maintainers to avoid problems with icon4py spack package.
- [x] **ICON4Py Tools**
- [x] Create a new package `icon4pytools` containing all the CLIs for integration of icon4py code into ICON.
- [x] Standardize the interfaces and help messages of the CLI tools to use a common format/language.
- [x] **Improve test suite**
- [x] Expand the `testutils` package with subpackages for containing helpers/utils for the test suite of each subpackage.
- [x] Create subpackages for `liskov`, `f2ser`, `icon4py`, and any other relevant subpackages.
- [x] Include relevant files for each subpackage in their respective `helpers.py` files.
- [x] **General Cleanup Work**
- [x] Ensure that each installable subpackage has its own README, and that each README is up to date.
- [x] Ensure that the root-level README of the repository is up to date.
- [x] Delete all obsolete or stale branches or PRs.
- [x] Complete the about section in the repository.
- [x] **MyPy Support**
- [x] Add MyPy to the QA pipeline to run on all source code.
- [x] Add MyPy to `precommit` to run on all source files for each subpackage.
- [x] Change the current repo structure to reflect the new MyPy invocation.
- [x] **Introduce CI for each subpackage**
- [x] Introduce CI pipeline (tox, qa) for icon4pytools and atm_dyn_iconam
- [x] **Introduce versioning**
- [x] Ensure there is a global `__version__` that is exported by each package.
- [x] Introduce `bumpversion` to bump the versions for each package.
- [ ] Open PR with refactored model code with new package `icon4py`
# TODO after green line is merged
- [ ] **Unit tests**
- [ ] Refactor all stencil unit tests to call the corresponding `field_operator` instead of the corresponding stencil program.
- [ ] **ICON4Py Model**
- [ ] Create a new package `icon4py` that contains all code related to the ICON4Py Model. Move the code
Sign in with Wallet
Connect another wallet