# Omnitrace - ENCCS Hackathon

* Reserve a GPU

* Load Omnitrace

ml rocm/5.3.3
source /cfs/klemming/home/g/gmarkoma/Public/omnitrace/1.7.4/share/omnitrace/setup-env.sh

* Allocate resources with `salloc`

* Check the various options and their values and also a second command for description

`srun -n 1 --gpus 1 omnitrace-avail --categories omnitrace`
`srun -n 1 --gpus 1 omnitrace-avail --categories omnitrace --brief --description`

* Create an Omnitrace configuration file with description per option

`srun -n 1 omnitrace-avail -G omnitrace_all.cfg --all`

* Declare to use this configuration file: `export OMNITRACE_CONFIG_FILE=/path/omnitrace_all.cfg`

* Get the file https://github.com/ROCm-Developer-Tools/HIP/tree/develop/samples/2_Cookbook/0_MatrixTranspose/MatrixTranspose.cpp

or `cp /project/project_465000388/exercises/AMD/MatrixTranspose.cpp .`
* Compile `hipcc --offload-arch=gfx90a -o MatrixTranspose MatrixTranspose.cpp`

* Execute the binary: `time srun -n 1 --gpus 1 ./MatrixTranspose` and check the duration

#### Dynamic instrumentation

* Execute dynamic instrumentation: `time srun –n 1 –-gpus 1 omnitrace -- ./MatrixTranspose` and check the duration
* Check what the binary calls and gets instrumented: `nm --demangle MatrixTranspose | egrep -i ' (t|u) '`
* Available functions to instrument: `srun -n 1 --gpus 1 omnitrace -v 1 --simulate --print-available functions -- ./MatrixTranspose`
* the simulate option means that it will not execute the binary

#### Binary rewriting (to be used with MPI codes and decreases overhead)
* Binary rewriting: `srun -n 1 --gpus 1 omnitrace -v -1 --print-available functions -o matrix.inst -- ./MatrixTranspose`
* We created a new instrumented binary called matrix.inst

* Executing the new instrumented binary: `time srun -n 1 --gpus 1 ./matrix.inst` and check the duration
* See the list of the instrumented GPU calls: `cat omnitrace-matrix.inst-output/TIMESTAMP/roctracer.txt`
#### Visualization

* Copy the `perfetto-trace.proto` to your laptop, open the web page https://ui.perfetto.dev/ click to open the trace and select the file

#### Hardware counters

* See a list of all the counters: `srun -n 1 --gpus 1 omnitrace-avail --all`
* Declare in your configuration file: `OMNITRACE_ROCM_EVENTS = GPUBusy,Wavefronts,VALUBusy,L2CacheHit,MemUnitBusy`
* Execute: `srun -n 1 --gpus 1 ./matrix.inst` and copy the perfetto file and visualize

#### Sampling

Activate in your configuration file `OMNITRACE_USE_SAMPLING = true` and `OMNITRACE_SAMPLING_FREQ = 100`, execute and visualize

#### Kernel timings

* Open the file `omnitrace-binary-output/timestamp/wall_clock.txt` (replace binary and timestamp with your information)
* In order to see the kernels gathered in your configuration file, make sure that `OMNITRACE_USE_TIMEMORY = true` and `OMNITRACE_FLAT_PROFILE = true`, execute the code and open again the file `omnitrace-binary-output/timestamp/wall_clock.txt`