- Shaped by: Christoph Müller, Nina Burgdorfer
- Appetite (FTEs, weeks): 2 weeks
- Developers: <!-- Filled in at the betting table unless someone is specifically required here -->

## Problem
Jenkins is not yet running on balfrin and no experiment relevant for operations is tested (like `mch_bench_r19b07_dev_sppt` as a minimal requirement), which means we will not notice if the experiments stop working on balfrin.

Additionally, we are accumulating so many variants of icon-exclaim (CPU, GPU, fused (verification/substitution/serialization), unfused (verification/substitution/serialization)) that not all of them can be tested daily, which has led to staleness of untested variants, for example for serialization.

As soon as we have optional components, planned for tracer advection and graupel, we might also want to switch those on/off to test their effects separately.

## Appetite
<!-- Explain how much time we want to spend and how that constrains the solution -->

## Solution
* Establish PR plan for Balfrin for icon4py
* Daily and PR jenkins plan, on tsa and daint, to test only the production critical variants, fused verification and fused substitution, with optional components switched off.
* Weekly jenkins plan, on tsa and daint, which tests all variants to make sure nothing goes stale.
* Establish daily, PR and weekly spack-based jenkins plans on balfrin derived from spack-based jenkins plan on daint, which test production relevant experiments (`sppt` or small icon1,icon2,kenda1 variants).
* This requires an ICON update of icon-exclaim, since the icon1, icon2 and kenda1 experiments do not run with icon-exclaim at the moment.

## Rabbit holes
<!-- Details about the solution worth calling out to avoid problems -->

## No-gos
<!-- Anything specifically excluded from the concept: functionality or use cases we intentionally aren’t covering to fit the ## appetite or make the problem tractable -->