# ICON4Py benchmark on Piz Daint

Make sure to always work on scratch file system. Working on home file system will affect the entire system performance, including other users on the machine. For build and test execution, make sure to allocate a compute node (never run from login node).

Piz Daint user guide: https://user.cscs.ch/access/running/piz_daint/

This environment is setup using Spack to build the GT4Py dependencies (boost, cmake and Python 3.10). The gcc compiler is already available as a module on Piz Daint and simply loaded (see step 1).

1. Load gcc compiler

module load gcc
export CC=gcc CXX=g++

2. Install Spack (one time setup)

cd $SCRATCH
mkdir repo
cd $SCRATCH/repo
git clone -c feature.manyFiles=true https://github.com/spack/spack.git
source spack/share/spack/setup-env.sh

3. Config spack environment and build dependencies on a cpu node (one time setup)

cd $SCRATCH/repo
srun -A <YOUR_ACCOUNT> -C mc -N 1 -t "02:00:00" --pty bash
spack env create gt4py
spack env activate -p gt4py
spack add boost cmake py-pip python@3.10
spack concretize
spack install
spack env deactivate
exit  # terminate slurm allocation on cpu node

4. Checkout icon4py repo (one time setup)

cd $SCRATCH/repo
git clone https://github.com/C2SM/icon4py.git

5. Create python environmnet (one time setup)

Note that we export `BOOST_ROOT` in order to build GT4Py stencils with the GTFN backend.

module load daint-gpu
module load gcc
export CC=gcc CXX=g++
source $SCRATCH/repo/spack/share/spack/setup-env.sh
cd $SCRATCH/repo/icon4py
srun -A <YOUR_ACCOUNT> -C gpu -N 1 -t "01:00:00" --pty bash
spack env activate -p gt4py
export BOOST_ROOT=$SPACK_ROOT/var/spack/environments/gt4py/.spack-env/view/include/boost
python -m venv .venv-py310
source .venv-py310/bin/activate
python -m pip install -r requirements-dev.txt 
python -m pip install cupy-cuda110
python -m pip install dace
pytest -s --benchmark-skip --backend=gtfn_gpu --grid=simple_grid model/atmosphere/diffusion/tests/diffusion_stencil_tests/test_calculate_nabla4.py
deactivate  # exit python environment
spack env deactivate
exit  # terminate slurm allocation on gpu node

6. Run benchmark

module load gcc
module load daint-gpu
source $SCRATCH/repo/spack/share/spack/setup-env.sh
cd $SCRATCH/repo/icon4py
srun -A <YOUR_ACCOUNT> -C gpu -N 1 -t "01:00:00" --pty bash
spack env activate -p gt4py
export BOOST_ROOT=$SPACK_ROOT/var/spack/environments/gt4py/.spack-env/view/include/boost
source .venv-py310/bin/activate
pytest -s -m 'not slow_tests' --benchmark-only --backend=gtfn_gpu --grid=simple_grid model/atmosphere/diffusion/tests/diffusion_stencil_tests
deactivate  # exit python environment
spack env deactivate
exit  # terminate slurm allocation on gpu node