---
# System prepended metadata

title: Diffusion granule integration - performance measurement

---


# Diffusion granule integration - performance measurement

## Introduction
This guide provides instructions on measuring the performance of the diffusion granule using the MCH R04B09 experiment on tsa. It includes information on integrating the diffusion granule into ICON, profiling performance, and important notes on experiments and environment variables.

### Relevant Resources
- **ICON Branch for Diffusion Granule Integration:** [GitHub Pull Request](https://github.com/C2SM/icon-exclaim/pull/280)
- **Setup Guide for tsa/balfrin Environment:** [HackMD Document](https://hackmd.io/lnbpB4CZSyemf5rQwdOYgw)
- **Optimized GT4Py Branch:** [GitHub Pull Request](https://github.com/GridTools/gt4py/pull/1536)
- **Python Diffusion Granule Overhead Profiling PR:** [GitHub Pull Request](https://github.com/C2SM/icon4py/pull/479)

## Granule Performance Measurement

### OpenACC Baseline (ICON-DSL)
- **Average Time per Timestep for Diffusion Run:** 0.00177s

### Python Diffusion Granule Called from ICON (Py2F)
- **Integration Method:** Py2F into ICON using the optimized gt4py branch.
- **Average Time per Timestep for Diffusion Run:** 0.00203s

### Python Diffusion Granule Called from Python (run_diffusion.py)
- **Method:** Using run_diffusion.py script with random data.
- **Execution:** On tsa compute node in interactive mode.
- **Average Time per Timestep for Diffusion Run:** 0.00164s

### Timing Diffusion Granule Runtime from Fortran
- **Flag for Profiling:** `--profile` in `py2f` tool.
- **Runtime of Diffusion Run Granule:** Approximately 0.00168s
- **Overhead Approximation:** 
  - 30% time in unpacking.
  - 70% time in creating gt4py fields (np_as_located_field).

## Granule Python Profiling using cProfile
Profiling is done using `cProfile` to understand time spent in Python code. The `.profile` file generated can be analyzed with `snakeviz`.

### Steps to Enable Profiling
1. **Embedding Profiling Functions:**
   - Embed `profile_enable` and `profile_disable` functions in addition to `diffusion_init` and `diffusion_run`.
   - Functions are located at: [diffusion.py](https://github.com/C2SM/icon4py/blob/main/tools/src/icon4pytools/py2fgen/wrappers/diffusion.py)
2. **Command to Run py2f and embedding profiler:**
   ```shell
   py2fgen icon4pytools.py2fgen.wrappers.diffusion diffusion_init,diffusion_run,profile_enable,profile_disable diffusion_plugin -b GPU
   ```
3. **Fortran ICON Code:**
   - Call `profile_enable()` and `profile_disable()` subroutines appropriately.
4. **Analyzing Profile Data:**
   - Use `snakeviz <.profile filename>` to visualize profiling data.

![Profiling Visualization](https://hackmd.io/_uploads/HkGZkwl8A.png)

## Other Important Notes

### Experiment Support
- **Supported Experiments:** mch_r04b09, exclaim_ape_ro4b05.
- **mch_r04b09 Specific:** Use `--limited-area` flag with py2f.
- **exclaim_ape_r04b05 Specific:** Omit `--limited-area` flag.
- **Post py2f Steps:** Copy new Fortran interface to ICON (src) and recompile.

### Environment Variables
Set the following environment variables in the run script:
- **ICON_GRID_LOC:** Path to the grid used by the experiment.
- **ICON4PY_LAM:** Set to `1` for MCH experiment, omit otherwise.
- **ICON4PY_BACKEND:** Set to `GPU` for GPU computation, default is `CPU`.

### GT4Py Optimizations
For optimal performance:
- Install gt4py in `_external_src` directory within icon4py.
- Use optimized branch: [GitHub Pull Request](https://github.com/GridTools/gt4py/pull/1536).
