[Blueline] CI for Blue Line on Balfrin and Santis

# [Blueline] CI for Blue Line on Balfrin and Santis  - Shaped by: Daniel - Appetite (FTEs, weeks): 1-2w - Developers: TBD ## Problem As developers of ICON-EXCLAIM, we want to ensure the correctness of the code while actively making changes. Currently, there is no consistent automated testing framework integrated with the main repositories, which makes it harder to detect regressions—especially for experiments running on the Balfrin and Santis systems. * A reliable continuous integration (CI) setup will help catch errors early and improve confidence in code changes. Beyond simply running tests, we also need a clear strategy for: * Defining setups that need to be tested, so that coverage is consistent across key experiment configurations. * Maintaining up-to-date testing infrastructure, ensuring tests continue to run correctly as the code evolves. * Deciding which versions and dependencies of ICON-EXCLAIM should be tested, to guarantee correctness across supported environments. This structured approach will allow us to detect regressions quickly, ensure reproducibility, and maintain high confidence in ongoing development. ## Appetite The goal is to implement a CI solution that can be completed within 1–2 weeks of effort. This timeline means the solution should primarily leverage existing CI infrastructure (such as CSCS-CI or DKRZ GitLab CI) rather than building a new pipeline from scratch. ## Solution We have identified two feasible options for implementing CI for ICON-EXCLAIM: 1. Enable CSCS-CI in ICON-EXCLAIM on GitHub * Needs to buildup CI infrastructure to run tests on both Santis but mainly Balfrin. * This approach keeps the project within the GitHub ecosystem and allows us to continue using the current repositories. 2. Use DKRZ GitLab CI * Use DKRZ’s already established CI pipelines by moving the repository from GitHub to GitLab. * This may involve forking the repository or coordinating with the ICON community to ensure smooth integration. * This approach may reduce maintenance overhead by relying on a CI system that is already well integrated with ICON environment. * This approach can be seen of chance to discuss with the ICON comunity about merging the blueline into icon main. ## Rabbit holes - Long discussions with the ICON community repository forks, and merge intentions can take significant time. Communication must be careful to manage expectations. - Any solution that requires building a completely new CI infrastructure would likely exceed the available appetite and is therefore considered out of scope. ## No-gos - Building a completely new CI pipeline from scratch using Jenkins or other systems. - Any solution that cannot be implemented within the 1–2 week appetite window. ## Progress - [ ] Preparation Task: - [x] Subtask merge: [spack-c2sm PR](https://github.com/C2SM/spack-c2sm/pull/1059) - [x] Subtask finish up: [icon-exclaim PR](https://github.com/C2SM/icon-exclaim/pull/384) on Santis - [x] Create uenv on Balfrin - [x] Subtask finish up: [icon-exclaim PR](https://github.com/C2SM/icon-exclaim/pull/384) on Balfrin - [x] Create uenv on Balfrin - [ ] Decide on Option 1 or 2 - [ ] Update icon main into icon-exclaim (shouldn't be needed, but is usefull) - [ ] Option 1: Setup CI in c2sm github - [ ] enable CSCS-CI in ICON-EXCLAIM on Santis - [ ] establish CSCS-CI in ICON-EXCLAIM on Balfrin - [ ] Option 2: Use CI in dkrz gitlab - [ ] Decide on posible fork with ICON-Comunity (please be carful with comunication, expactation management, and merge intentins), either use icon-cscs or a new icon-exclaim - [ ] Move icon-exclaim from github to gitlab, see: [[Blueline] Prepare to merge icon-exclaim upstream](https://hackmd.io/iQbBDsiWT2avZHGEwWAb3A?both) - [ ] Enable CI - [ ] Task revise testing - [ ] identify missing nightly tests, CI tests - [ ] add torus grid to testing-input-data - [ ] Use at least small global experiments exp.exclaim_uncoupled_R02B05L120 to benchmark on santis and the MCH production experiments to benchmark on balfrin (fix current bencher setup to have these experiments) ## Occured problems ``` ==> Error: build_directory ``` fix be swiping env and concretize spack develp, concretize -f