### Reliable Flamegraphs for Substrate-Based Binaries

This note documents how to profile a *substrate-based* node and generate nice flame graphs. Some tweaks are required for building process: all *obvious* technical details are left here for future investigators.

- Make sure this [PR](https://github.com/paritytech/polkadot-sdk/pull/9821) is on your branch, if not cherry-pick support for [`perfmap` in wasmtime](https://github.com/paritytech/polkadot-sdk/commit/8978c005de6631dce20e204380bb43149127cdce),
- enable *debug* symbols for the build (not sure if also needed for runtime - needs to be confirmed) in `Cargo.toml`:

    [profile.release]
    debug = true
    
    [profile.production]
    debug = true

- The default DWARF-based stack-unwinding method used by perf is not always reliable. More accurate results can be achieved by using *frame pointer*. This requires building the codebase with `force-frame-pointers=yes` compiler flag. Rebuild the benchmark (or other binary, e.g. the node):

    export RUSTFLAGS="-C force-frame-pointers=yes"
    cargo bench --bench block_production

_Note_: using this flag may slightly increase the binary size and also have a small impact on performance. In most cases it should be negligible. Some references: [[1](https://rfc.archlinux.page/0026-fno-omit-frame-pointer/)], [[2](https://ubuntu.com/blog/ubuntu-performance-engineering-with-frame-pointers-by-default)],
- Before profiling: some linux distributions may require following setup to allow `perf` tool accessing kernel interfaces for profiling:

    # note: some OSs may require this:
    sysctl -w kernel.perf_event_paranoid=0
    sysctl -w kernel.kptr_restrict=0

This can be checked by profiling any binary, e.g. `perf record ls`.
- Profile the binary:

    # /tmp/perf-<pid>.map will be created
    export WASMTIME_PROFILING_STRATEGY=perfmap
    perf record -F 999 -g -- /path-to/target/release/deps/block_production-xxxx --bench

When working with *zombienet* the node execution needs to be wrapped with `perf record` command. This can be done by creating a simple wrapper script (`polkadot-parachain-wrapper`) for target node binary. This script shall be used in zombienet config as the node *command*:

    #!/bin/bash -x
    export WASMTIME_PROFILING_STRATEGY=perfmap
    # /tmp/perf-<pid>.map and perd.data files will be created
    perf record -F 999 -g --output=perf.data -- /path/to/node "$@"

- Converting recorded results (`perf.data` and `/tmp/perf-<pid>.map` files) to *flame graph*:

    #pre-requisites:
    cargo install addr2line --features=bin
    git clone https://github.com/brendangregg/FlameGraph.git

    #processing recorded perf.data (/tmp/perf-<pid>.map created by wasmtime will also be read by perf)
    perf script | rustfilt > perf.script
    FlameGraph/stackcollapse-perf.pl perf.script > perf.folded
    FlameGraph/flamegraph.pl perf.folded > flamegraph.svg

- To make flamegraph even better, `FlameGraph/stackcollapse-perf.pl` can be patched to improve generic-heavy rust function names:

    diff --git a/stackcollapse-perf.pl b/stackcollapse-perf.pl
    index 3ff39bf..d23dd55 100755
    --- a/stackcollapse-perf.pl
    +++ b/stackcollapse-perf.pl
    @@ -348,7 +348,10 @@ while (defined($_ = <>)) {
                        # "net/http.(*Client).Do"), so everything after the first open
                        # paren (that is not part of an "(anonymous namespace)") is
                        # just noise.
    -                   $func =~ s/\((?!anonymous namespace\)).*//;
    +                   # Preserve Rust template parameters in angle brackets
    +                   if ($func !~ /<.*>/) {
    +                       $func =~ s/\((?!anonymous namespace\)).*//;
    +                   }
                    }
                    # now tidy this horrible thing:
                    # 13a80b608e0a RegExp:[&<>\"\'] (/tmp/perf-7539.map)

- Helper script for generating *svg*. Useful when lot of runs are made. Not really needed, just improves a flow a bit.

    #!/bin/bash -x

    if [ "$#" -ne 1 ]; then
        echo "Error: name shall be given"
        exit 1
    fi

    DIR=`mktemp -d -p . -t log---$1--XXXXX`

    echo $DIR
    ls -al $DIR

    mv perf.data $DIR

    pushd $DIR
    perf script | rustfilt > perf.script
    ../FlameGraph/stackcollapse-perf.pl perf.script > perf.folded
    ../FlameGraph/flamegraph.pl perf.folded > flamegraph.svg
    popd

    ls -al $DIR/flamegraph.svg
    /opt/google/chrome/chrome $DIR/flamegraph.svg