---
# System prepended metadata

title: Optimizing SDFG of `fv3core` Acoustic Substep
tags: [dace, cycle 4]

---

# Optimizing SDFG of `fv3core` Acoustic Substep 
###### tags: `cycle 4` `dace` 

In the past cycle, we were able to get an SDFG of the acoustic substep of fv3core. The goal of this project is to measure and improve the runtime of the substep on Piz Daint nodes, targeting the fortran reference implementation's performance or an improvement over it.

Unfortunately, there is no directly equivalent baseline performance available for just the acoustic standalone without halo exchanges. We therefore define the following points of reference regarding performance:
* Time spent in the acoustic substep and its largest non-communicating parts (like d_sw), when runing the full FV3 dycore Fortran implementation
* Python-driven GT4Py code with the best CPU backend available
* Python-driven GT4Py code with the best GPU backend available
* Based on the timings for the full dycore, we can guess a conversion factor from Python-driven to fortran codes.


## Tasks
1. Establish performance baselines on CPUs and GPUs of daint-gpu nodes:
   * establish comparable timings in python
   * tune deployments (env vars, thread bindings, ...)
   * representative measurements in fv3core Fortran
2. Experiment with performance, by means of existing tools, for example
   * at DaceProgram scope: dace-builtin transformations
   * at gtscript stencil scope: improving OIR transformations, improving OIR node expansion
   
   To learn about the effectiveness of transformations, this can be approached in two substeps
   a. manual transformation candidate selection, selection of OIR optimizations
   b. fully automated pipeline, indiscriminately or based on performance heuristics
   
   **Milestone**: At this point we expect to beat native (non-jit) python running gt4py stencils with any CPU backend and get close to a similar speedup like Fortran over python with gt4py on the full dycore.
   
3. Should the results of 2. show the possibility to significant further improvements, the development of new 
   * new "low-level" trafos (possible reuse from 2020 GTBench?, mostly stencil-scope)
   * new OIR transformations (stencil-scope only)
   
   **Milestone**: Faster than Fortran in those parts that are comparable (i.e. not broken up by halo exchanges) such as d_sw. For GPU, we target a factor of 2-4 over fortran here. 

## Goals
* Demonstrate runtimes better than any python+gt4py performance on CPU, based on a fully automated code generation and transformation scheme
* Depending on resource investment, Fortran perfromance can be targeted and/or a GPU target hardware can be supported as well.

### No-Goals
* Multi-node optimization / scalability
* Optimizing Parsing / Codegen times.

## Appetite
* 1 Performance engineer full cycle to get the CPU performance to the first Milestone.
* With an additional full cycle FTE GPU can be targeted 
* Another half cycle FTE can be invested for CPU and GPU each, to get to get to the second Milestone.
We further rely on the Fortran timings being provided by vulcan. 