# Analyzing log with AMD_LOG_LEVEL=3

Filter only hipMemcpyDtoH lines, like: `cat out | grep hipMemcpyDtoH > dtoh`

You will have:

:3:hip_memory.cpp           :1202: 845477829515 us: 91967: [tid:0x14aa96eaa080] hipMemcpyDtoH ( 0x7ffce43712e8, 0x14a74d3d8000, 121616 )
:3:hip_memory.cpp           :1204: 845477829560 us: 91967: [tid:0x14aa96eaa080] hipMemcpyDtoH: Returned hipSuccess : : duration: 45 us
...

You can make a sum of bytes using:

cat dtoh | grep 'hipMemcpyDtoH (' | awk '{sum += $11} END {print sum/1024/1024}'

Or sum of durations using:

cat dtoh | grep 'hipMemcpyDtoH:' | awk '{sum += $13} END {print sum/1024/1024}'

Similar things for HtoD.

Besides making a sum, we can do more things with this data, like making a frequency histogram or whatever.

# Analyzing log for CRAY_ACC_NOTIFY=1

Filter only "ACC: ... to host" lines, like: `cat out | grep 'ACC:.*to host' | grep -v 'to host 0 bytes' > htod`

You will have:

ACC: End transfer (to acc 0 bytes, to host 121712 bytes)
ACC:       copy to host <internal> (233472 bytes)
ACC:       copy to host <internal> (20545536 bytes)
ACC:       copy to host <internal> (466944 bytes)
ACC:       copy to host <internal> (233472 bytes)
ACC:       copy to host <internal> (233472 bytes)
ACC:       copy to host <internal> (20545536 bytes)
ACC:       copy to host <internal> (466944 bytes)
ACC:       copy to host <internal> (233472 bytes)
ACC:       copy to host <internal> (233472 bytes)

# Separate log file

We can find the line numbers in the log for different kernel executions: just grep for `ACC: Execute kernel apl_alaro_radiation_parallel_$ck_L1006_4` in the log, printing the line numbers:

$ grep -n 'ACC: Execute kernel apl_alaro_radiation_parallel_$ck_L1006_4' CRAY_ACC_1_AMD_3_log.err | awk -F: '{print $1}'
46843
143160
328291
513456
698627
883798
1068969
1254140
1439311
1624482
1809653
1994828

Then with `tail -n` and `head -n` combination you can cut the part of logs correspponding to a single timestep, or do any other cuts.