# ICON-blue-alps [TOC] ## Santis ### Common instructions for building and running on Santis 1. Create access token on gitlab.dkrz.de and github.com: Santis is currently behind a firewall - Then you can do the following to store credentials in plain-text in `~/.git-credentials` the first time you are asked for password. ```bash~ git config --global credential.helper store ``` 2. Use Ben's magic squashfs to get the ICON environment. This is required before building and **also before launching run scripts**. ``` srun -n1 -t180 --pty --uenv=/bret/scratch/cscs/bcumming/images/icon-dsl-1.squashfs bash ``` 3. Use the token to clone with https (git@ will not work from Santis) 4. To be able to login to running compute nodes on Santis, you need to forward user agent at time of login ``` ssh -A santis ``` 5. When successfully running on compute nodes, you can login to the specific compute node and then do `nvidia-smi` to check GPU usage. ---------------- ### ICON-NWP #### Build Instructions (GPU) Building ICON-NWP on Santis (should this go here??) ```bash! git clone --recursive -b abishekg7_santis https://username:<your token here>@gitlab.dkrz.de/icon/icon-nwp.git icon-nwp-for-santis (or if you opted to store your credentials in `~/.git-credentials`) git clone --recursive -b abishekg7_santis https://@gitlab.dkrz.de/icon/icon-nwp.git icon-nwp-for-santis cd icon-nwp-for-santis ./setup_gpu.sh # icon-nwp-for-santis should contain the latest version of this script to configure and build ICON + generate run script ``` This starts the long process of building ICON (can take up to 10 minutes) #### Build Instructions (CPU) ```bash! git clone --recursive -b abishekg7_santis https://username:<your token here>@gitlab.dkrz.de/icon/icon-nwp.git icon-nwp-for-santis cd icon-nwp-for-santis ./setup_cpu.sh ``` - Need to remove GPU specific `FCFLAGS` from the configure file. Removed: - ` -acc=verystrict -gpu=cc90 -Minfo=accel` - `--enable-gpu=openacc+cuda` - `--enable-mpi-gpu` - `-L${CUDA_HOME}/lib64, -lcudart, -Wl,--as-needed` - `CUDAARHCS=` - `NVCFLAGS=` Added: - `--disable-gpu` #### Run instructions - make sure user environment is loaded in current shell, by using `srun` command above - For icon-nwp, we can set `nblocks_c=1`, so theres only one block of cells, this sets nproma properly, and improves performance. - For full-grid radiation, use `nblocks_sub=1` instead of `nproma_sub`, for performance reasons - For reduced-grid radiation, `nproma_sub` needs to be tuned. More details [here](https://gitlab.dkrz.de/icon/wiki/-/wikis/GPU-development/GPU-Implementation#how-to-choose-nproma_sub-for-ecrad-for-simulations-without-atmospheric-nesting) ---------------- ### ICON-DSL - We use the `grace-hopper` branch of icon-exclaim: https://github.com/C2SM/icon-exclaim/tree/grace-hopper #### Build Instructions (DSL) We currently support building only the DSL version on Santis. ```bash! cd $SCRATCH mkdir icondsl_build && cd icondsl_build # Copy env.sh, install_dependencies.sh, setup_santis_dsl.sh into this dir ./install_dependencies.sh ./setup_santis_dsl.sh ``` icon-dsl builds take muuch longer than icon-nwp #### Run instructions - make sure user environment is loaded in current shell, by using `srun` command above - For icon-dsl, we need to manually set `nproma` such that all edges fit inside one block. Ask others for more details. ## Vial ## TODO - [ ] Consolidate `setup_gpu.sh` and `setup_cpu.sh` into one `setup_santis.sh` script which accepts a `cpu` or `gpu` argument allowing building of both modes.