---
# System prepended metadata

title: Jenkins setup for ICON-exclaim master
tags: [functional cycle 10]

---

# Jenkins setup for ICON-exclaim master

###### tags: `functional cycle 10`

Appetite: 1/2 cycle

Developer: Abishek 

## Goals

Implement a Jenkins CI pipeline for ICON-EXCLAIM/master, similar to the C2SM/ICON or the ICON-EXCLAIM/icon-dsl setups.

## Known tasks and progress

Working in personal fork of ICON-EXCLAIM/master for now

1. Submodules
    - [x] Update submodules to reference internal C2SM repositories, avoiding DKRZ references, similar to C2SM/icon
     
2. Jenkins configuration
    - [x] Added a jenkins directory, containing the Jenkins files and supporting scripts ~~(in the scripts subdirectory)~~.
    - [x] `jenkins/Jenkinsfile_daily` and `jenkins/Jenkinsfile_PR` have been adapted from icon-dsl
    - [x] Scripts in jenkins/scripts are largely similar to C2SM/icon scripts/jenkins_scripts but I removed the jenkins.sh wrapper, and Sam's new common.sh is used to setup some environment variables instead.
3. Icon config and build
    - [x] ~~Adding new config scripts to the config/cscs/c2sm directory. This may or may not be required going forward.~~ 
    - [x] Now using config scripts from config/cscs directory to remain consistent with icon-nwp.
    - [x] Using spack to set library paths required by configure scripts
    - [ ] Modify remaining config scripts, that currently hard-code dependecy paths, to use spack
    - [ ] Test build process with cray/cce on Daint. 
4. Probtest
    - [x] `setup_probtest` pipeline stage queries for the current hash of probest (using BB scripts) and then installs a fresh venv + prerequisites, similar to the icon-dsl scripts.
    - [x] Generate new probtest references for daint_cpu_pgi and daint_gpu_nvidia. Currently stored in the icon-test-references subdirectory, but may need to move this to a separate repository.
    - [x] Obtain icon-test-references from DKRZ gitlab instead of generating internally. Occurs during `setup_probtest` stage.
    - [ ] Probtest comparison fails for some currently excluded tests. Investigate if related to code being out of sync.
    - [ ] Cleanup tolerance.sh or remove? 
5. Testing framework
    - [x] Testing framework remains close to C2SM/icon and DWD/icon-nwp. ~~Copied the checksuite.icon-c2sm directory into run, from C2SM/icon. And modified some tests to include tolerance testing (in the MD env variable).~~
    - [x] Using checksuite.icon-dev and removing checksuite.icon-c2sm
    - [x] Testlists are stored in jenkins/scripts/test.daint.*
    - [ ] Integrate Jonas' PR11 - Use spack/test.py to mimick BuildBot testsuite. 

6. Jenkins PR plan
    - [x] Implemented a Jenkins plan for PR
    - [x] Avoid conflicts with icon-dsl PR plan by specifying target branches to white list (master) and exclude (icon-dsl).
7. Cleanup and finalize
    - [ ] Merge personal fork with ICON-EXCLAIM/master
    

## Issues

1. `icon-exclaim/master` is out of sync with `icon-nwp/master`. Not every test will pass until this sync. Christoph is presently working on syncing `icon-exclaim/icon-dsl` with `icon-nwp/master`. Should we wait till Oct/Nov when `icon-exclaim/icon-dsl` is merged with `icon-exclaim/master`, or try to indepdently sync with `icon-nwp/master` now?