# Toolchain Customization Flags (moved from cycle 15)

###### tags: `cycle 15`

- Shaped by: @ricoh
- Appetite (FTEs, weeks): 2 FTE, 1 week
- Developers:
- Available for discussion/review: @egparedes

## Problem

The OTF toolchain transforms and compiles GT4Py code into executable python code (and other artifacts along the way). The one currently existing workflow for this is the one for GTFN (GridTools Functional). The entry point file for this is `next.program_processors.runners.gtfn_cpu`, the parts that are meant to be reusable by (future) other workflows are under `next.otf`. Each part of a workflow may have configuration options.

These options currently come in three flavours:

- global variables (example: `otf.compilation.cache._persistent_cache_dir_path`)
- constructor arguments to a workflow or subworkflow. Example:

    # GTFN with session or persistent caching
    otf.recipes.OTFCompileWorkflow(
        ...
        compilation=otf.compiler.Compiler(
            cache_strategy=cache.Strategy.<SESSION | PERSISTENT>
        )
    )

Each has a reason to exist and models what the option does most closely but from a user perspective there are downsides to having to work with all three.

**Global variables:** need to be changed in code. This means changing them for debugging in a feature branch will require you to undo that change before merging. It also means that running a test or set of test methods with a changed value has all the problems associated with monkey patching plus keeping track of global state to undo the monkey patching. Simply reading from an environment variable at startup (if it is defined) can fix the first problem but not the second.

**Workflow constructor arguments** are great to keep the configuration logic where it belongs. Combined with `otf.workflow.replace` they make it easy to change workflow configuration for isolated parts of code (like tests). The difficulty comes with changing the defaults based on user configuration (environment variables) or for temporary debugging.

Note that pre-configured ready to use workflows are also global variables (see `program_processors.runners.gtfn_cpu`)

**Workflow structure** can be used to for example optionally skip (parts of) the workflow conditionally (see `otf.workflow.CachedStep`).

### Known use cases

- Run tests for `gtfn_cpu` using `cache_strategy=cache.Strategy.PERSISTENT` or `SESSION` without touching any version controlled files. Tests that excplicitly set the `cache_strategy=...` already should be excluded from this behaviour.
- Same for switching all ITIR transforms on / off in the translation step of `gtfn_cpu`
- Change the place to which compilation artefacts are cached without touching any version controlled files as well as for specific tests.
- Change CMake build type in the same manner.

Note that this could also be implemented in terms of "debug levels"

### Potential use cases

Recognize potential usecases by:

- they are inside the toolchain
- they are fairly common
- the configuration variable in question is not specific to any workflow

## Appetite

- Quality of life for developers is affected
- This would break the ground for streamlining the end-user interface (specifying the whole workflow from python function to end result in the decorator, with unified configuration method)

## Solution

Before implementing any of the usecases it will be crucial to solicit feedback from at least two other GT4Py devs.

### Steps

1. Schedule a discussion for max 1-2 days after the start of the project to solicit feedback from at least two other devs. Use Standup + Slack channel to announce it.
2. Propose a design based on below considerations, refine with the help of feedback.
3. Document your choices and designs in an ADR.
4. Implement the known use cases (keep the ADR updated in case of changes), then look for more if there is time.

### Design considerations

A solution must provide the following
- There should be one obvious way to read user config.
> This could for example be through environment variables.
- There should be one and only one obvious place where user config is read at import time.
> Something like `gt4py.next.settings` or `gt4py.next.otf.settings`.
- There needs to be way to allow changing the defaults for a specific context in code.
> [Context Variables](https://docs.python.org/3/library/contextvars.html) should be a good starting point and might turn out to be good enough.
- The current interface for building (and therefore configuring) workflows from steps can be kept but should be considered low-level.

A good solution is characterized by
- it's easy to use
- it's easy to support from the point of view of a toolchain implementer
- it doesn't break a toolchain that doesn't explicitly support it. Ideally it notifies the user when they are using a toolchain that does not support their config. Example: `roundtrip` backend does not support compilation caching.

## Rabbit holes

- Allowing to override workflow configuration made in client code could in theory be interesting but would likely be extremely hard to design and implement in a non-surprising way.
- Trying to design a comprehensive interface for any potential workflow to support any and all configuration would probably lead too far

## No-gos

- Do not implement multiple ways for the user to communicate configuration (env vars + config files for example), this project is too small for that.
- Adding features like logging or a pass manager throughout the toolchain are explicitly not part of this project.

## Progress

- [X] PoC
- PR: https://github.com/GridTools/gt4py/pull/1438
- Collab: https://colab.research.google.com/drive/1LUQyKFtlYVmbMeH1UvKxGpFez_2h4yUb?usp=sharing
- [X] Experiment with warnings for ignored user configuration
- [x] ADR
- [x] Tests
- [x] User documentation