HackMD
Profiling Tools
Rocprof
- Get the exercise:
https://github.com/ROCm-Developer-Tools/HIP-Examples/tree/master/mini-nbody
- Compile and run the code
Can compile and run all with
Or just run compile and run one case
- Note that we can find the compile line and the executable name by checking the file
HIP-nbody-orig.shor by watching the output from running the script.
-
The binary is called
nbody-orig -
Use rocprof with
--stats
Files with the prefix results are created
- Check the files results.csv
You can see information for each kernel call with their duration
- Check the statistics result file, one line per kernel
- Profile the HIP calls with
--hip-trace
Now we have new files with the hip in their name like below, check the file results.hip_stats.csv
- Profile also the HSA API with the
--hsa-trace
- See the content of the file
results.hsa_stats.csv
- Download the results.json file on your laptop
From your laptop:
scp username@lumi.csc.fi:/path/results.json .
- Visit the web page:
-
Click on the top left menu, "Open Trace File on th eleft top"
-
Select the file results.json
Zoom in/out: W/S
Move left/right: A/D
Read about the counters: vim /opt/rocm/rocprofiler/lib/gfx_metrics.xml
- Create a file with the contents:
- Execute with using the counters
- Contents of the rocprof_counters.csv file
Omnitrace
We have made special builds of the Omnitools, omnitrace and omniperf for use in the exercises
- Reserve a GPU
salloc -p gpu --gpus 1 -t 00:40:00
- Load Omnitrace
Load the module or declare PATH and LD_LIBRARY_PATH
- Check the various options and their values and also a second command for description
srun -n 1 --gpus 1 omnitrace-avail --categories omnitrace
srun -n 1 --gpus 1 omnitrace-avail --categories omnitrace --brief --description
- Create an Omnitrace configuration file with description per option
srun -n 1 omnitrace-avail -G omnitrace_all.cfg --all
-
Declare to use this configuration file:
export OMNITRACE_CONFIG_FILE=/path/omnitrace_all.cfg -
Get the file https://github.com/ROCm-Developer-Tools/HIP/tree/develop/samples/2_Cookbook/0_MatrixTranspose/MatrixTranspose.cpp
-
Compile
hipcc -o MatrixTranspose MatrixTranspose.cpp -
May need to add the GPU architecture
- Execute the binary:
time srun -n 1 --gpus 1 ./MatrixTransposeand check the duration
Dynamic instrumentation
- Execute dynamic instrumentation:
time srun –n 1 –gpus 1 omnitrace -- ./MatrixTransposeand check the duration - Check what the binary calls and gets instrumented:
nm --demangle MatrixTranspose | egrep -i ' (t|u) ' - Available functions to instrument:
srun -n 1 --gpus 1 omnitrace -v -1 --simulate --print-available functions -- ./MatrixTranspose- the simulate option means that it will not execute the binary
Binary rewriting
-
Binary rewriting:
srun -n 1 --gpus 1 omnitrace -v -1 --print-available functions -o matrix.inst -- ./MatrixTranspose- We created a new instrumented binary called matrix.inst
-
Executing the new instrumented binary:
time srun -n 1 --gpus 1 ./matrix.instand check the duration -
See the list of the instrumented GPU calls:
cat omnitrace-matrix.inst-output/TIMESTAMP/roctracer.txt
Visualization
- Copy the
perfetto-trace.prototo your laptop, open the web page https://ui.perfetto.dev/ click to open the trace and select the file
Hardware counters
- See a list of all the counters:
srun -n 1 --gpus 1 omnitrace-avail --all - Declare in your configuration file:
OMNITRACE_ROCM_EVENTS = GPUBusy,Wavefronts,VALUBusy,L2CacheHit,MemUnitBusy - Execute:
srun -n 1 --gpus 1 ./matrix.instand copy the perfetto file and visualize
Sampling
Activate in your configuration file OMNITRACE_USE_SAMPLING = true and OMNITRACE_SAMPLING_FREQ = 100, execute and visualize
Kernel timings
- Open the file
omnitrace-binary-output/timestamp/wall_clock.txt(replace binary and timestamp with your information) - In order to see the kernels gathered in your configuration file, make sure that
OMNITRACE_USE_TIMEMORY = trueandOMNITRACE_FLAT_PROFILE = true, execute the code and open again the fileomnitrace-binary-output/timestamp/wall_clock.txt
Omniperf
- Lod Omniperf:
- Reserve a GPU, compile the exercise and execute Omniperf, observe how many times the code is executed
-
Run
srun -n 1 --gpus 1 omniperf profile -hto see all the options -
Now is created a workload in the directory workloads with the name vcopy_all (the argument of the -n). So, we can analyze it
- If you want to only roofline analysis, then execute:
srun -n 1 --gpus 1 omniperf profile -n vcopy_all --roof-only -- ./vcopy 1048576 256
There is no need for srun to analyze but we want to avoid everybody to use the login node. Explore the file vcopy_analyze.txt
- We can select specific IP Blocks, like:
But you need to know the code of the IP Block
- If you have installed Omniperf on your laptop (no ROCm required for analysis) then you can download the data and execute:
- Open the web page: http://IP:8050/ The IP will be displayed in the output
