---
# System prepended metadata

title: ICON-blue-alps

---

# ICON-blue-alps

[TOC]

## Santis

### Common instructions for building and running on Santis

1. Create access token on gitlab.dkrz.de and github.com: Santis is currently behind a firewall
    - Then you can do the following to store credentials in plain-text in `~/.git-credentials` the first time you are asked for password. 
    ```bash~
    git config --global credential.helper store
    ```

2. Use Ben's magic squashfs to get the ICON environment. This is required before building and **also before launching run scripts**.

```
srun -n1 -t180 --pty --uenv=/bret/scratch/cscs/bcumming/images/icon-dsl-1.squashfs bash
```
3. Use the token to clone with https (git@ will not work from Santis) 

4. To be able to login to running compute nodes on Santis, you need to forward user agent at time of login
```
ssh -A santis
```

5. When successfully running on compute nodes, you can login to the specific compute node and then do `nvidia-smi` to check GPU usage. 

----------------
### ICON-NWP

#### Build Instructions (GPU)

Building ICON-NWP on Santis (should this go here??)



```bash!
git clone --recursive -b abishekg7_santis https://username:<your token here>@gitlab.dkrz.de/icon/icon-nwp.git icon-nwp-for-santis

(or if you opted to store your credentials in `~/.git-credentials`)

git clone --recursive -b abishekg7_santis https://@gitlab.dkrz.de/icon/icon-nwp.git icon-nwp-for-santis
    
cd icon-nwp-for-santis
    
./setup_gpu.sh   #  icon-nwp-for-santis should contain the latest version of this script to configure and build ICON + generate run script
```

This starts the long process of building ICON (can take up to 10 minutes)

#### Build Instructions (CPU)

```bash!
git clone --recursive -b abishekg7_santis https://username:<your token here>@gitlab.dkrz.de/icon/icon-nwp.git icon-nwp-for-santis
    
cd icon-nwp-for-santis

./setup_cpu.sh
```

- Need to remove GPU specific `FCFLAGS` from the configure file.

Removed:

- ` -acc=verystrict -gpu=cc90 -Minfo=accel`
- `--enable-gpu=openacc+cuda`
- `--enable-mpi-gpu` 
- `-L${CUDA_HOME}/lib64, -lcudart, -Wl,--as-needed`
- `CUDAARHCS=`
- `NVCFLAGS=`

Added:
- `--disable-gpu`

#### Run instructions
- make sure user environment is loaded in current shell, by using `srun` command above
- For icon-nwp, we can set `nblocks_c=1`, so theres only one block of cells, this sets nproma properly, and improves performance.
- For full-grid radiation, use `nblocks_sub=1` instead of `nproma_sub`, for performance reasons
- For reduced-grid radiation, `nproma_sub` needs to be tuned. More details [here](https://gitlab.dkrz.de/icon/wiki/-/wikis/GPU-development/GPU-Implementation#how-to-choose-nproma_sub-for-ecrad-for-simulations-without-atmospheric-nesting)    

----------------
### ICON-DSL

- We use the `grace-hopper` branch of icon-exclaim: https://github.com/C2SM/icon-exclaim/tree/grace-hopper
#### Build Instructions (DSL)

We currently support building only the DSL version on Santis. 



```bash!

cd $SCRATCH

mkdir icondsl_build && cd icondsl_build

# Copy env.sh, install_dependencies.sh, setup_santis_dsl.sh into this dir

./install_dependencies.sh

./setup_santis_dsl.sh

```

icon-dsl builds take muuch longer than icon-nwp

#### Run instructions
- make sure user environment is loaded in current shell, by using `srun` command above
- For icon-dsl, we need to manually set `nproma` such that all edges fit inside one block. Ask others for more details.

## Vial


## TODO

- [ ] Consolidate `setup_gpu.sh` and `setup_cpu.sh` into one `setup_santis.sh` script which accepts a `cpu` or `gpu` argument allowing building of both modes.
