# OCaml 5 Debugging Guide

Do you have issues with OCaml5 not working as expected?
Here we share some of the tips and tricks that the OCaml Multicore developers have benefitted from in debugging OCaml5 applications, along with tools to help test and verify OCaml5 code.

## Tools for performance debugging

### `htop`

Simply running `htop` can be useful to understand if your parallel code utilizes all your CPUs in parallel.

[ screenshot ]

Here one can see that all 4 cores are utilized fully when running the multicore OCaml5 program.

### `perf`

`sudo perf top` goes one step further and reports a live profile of where time is spent in your multicore program. This can be handy to identify hot spots in your code base. If you run it as `sudo perf top -p <pid>` you get a live profile for the OCaml process with a specific process id.

For a more serious performance analysis `perf` also lets you record the observed events with `perf record`.

`perf` supports multiple call-graph modes:
- `dwarf`
- `framepointer`
- `lbr` (on recent Intel processors)
- ...

More `perf` resources:
- More `perf` tips from Sadiq: https://twitter.com/sadiqj/status/1240386091569602562
- https://www.brendangregg.com/perf.html

Note: `perf` is Linux only

### `Runtime_events`

OCaml5 comes with `Runtime_events` - performant program events written to a ring-buffer for consumption by user tools. There are even [future plans](https://github.com/ocaml/ocaml/pull/11474) to extend `Runtime_events` with programmatic, custom events.

### `olly`

[olly](https://github.com/sadiqj/runtime_events_tools#olly) is a user tool to record and consume `Runtime_events`. With `olly` you can quickly get an idea of, e.g., the latency caused by garbage collection in your OCaml5 program.

### `ocamlprof`

[ocamlprof](https://v2.ocaml.org/manual/profil.html) is a profiler for OCaml, counting how many times a function is called, how often a branch is taken, etc. This information can be useful to get an idea of the "hot spots" where most execution time is spent.

### `statmemprof`

statmemprof is a compiler extension for statistical memory profiling of OCaml programs.

Note: This only works on 4.14 for now.

### `magic-trace`

[magic-trace](https://github.com/janestreet/magic-trace) is a tool to collect, visualize, and explore what a program is spending its CPU time doing. It builds on statmemprof.

Note: This only works on 4.14 for now (?)

### `memtrace`

To debug memory usage [memtrace](https://github.com/janestreet/memtrace) is handy to learn how the live memory evolves and pinpoint which functions may be responsible for increasing it.

Note: This only works on 4.14 for now. (Is this subsumed by run_time_events and olly?)

### `ocaml-probes`

[ocaml-probes](https://github.com/janestreet/ocaml-probes) can be used to inspect behaviour of a running problem, e.g., for performance debugging or logging.

## General debugging tools

If your code interacts with C there's a risk of crashes and heap corruption.

### Link your code against the debug runtime

By default `dune` will include debugging information (`-g`) when asked to build your OCaml project. This may be sufficient to debug the issue you are tackling.

OCaml also has a "debug runtime" that performs a range of additional `assert` checks. This means it will often fail earlier and closer to the root cause of the problem. You can use the following lines in your dune file to link against the debug runtime:

 (ocamlc_flags -g)
 (ocamlopt_flags -g)
 (link_flags -runtime-variant=d -g)

### Heap verification

### `rr`

`rr` is a replayable debugger - and a modern successor to `gdb`.

Note: `rr` runs concurrently, not in parallel to increase reproducability. Hence bugs triggered only by true parallelism may not be reproducable under `rr`.

Additional tips from KC: https://kcsrk.info/ocaml/multicore/rr/2019/04/28/0000-rr-debugging/

### `gdb` / `lldb`

A good old debugger such as `gdb` or `lldb` can be useful in case an issue is not reproducable under `rr`.

[ printing a parallel backtrace under both ]

Note: Like in OCaml4 and previous versions, OCaml values are not printed natively under these debuggers. Instead, `gdb` and `lldb` lets you inspect the representation of OCaml values.

### Study the produced code

Sometimes it can be useful to inspect the produced code to understand why a piece of code behaves in a certain way.
The flags `-dlambda`, `-dcmm`, ... can be useful to inspect the code at various stages through the compiler.
Note, it is generally easier to identify code in a function by searching for the function name compared to identifying code resulting from top-level expressions.

### `printf`

Inserting print-outs is a universal debugging aid used by many developers.
This approach can also be useful in multicore OCaml, but beware:
- In OCaml, a call to `Printf.printf` should preferably be flushed (you can use `Printf.printf "...\n%!"`) to avoid output getting stuck in an output buffer and leaving you chasing down the wrong path.
- Printing to the same console risks introducing output races, meaning the output get can interleaved and garbled - it may still be relevant though.

## Tools for debugging `Effect`s and `Eio`

### `ocamldebug`

[`ocamldebug`](https://v2.ocaml.org/manual/debugger.html) is a debugger for OCaml. It works over bytecode programs and has the features you would expect: breakpoints, forward and backward single stepping, printing variables, etc.

`ocamldebug` works with `Effect`s but not with multiple `Thread`s (or `Domain`s?): https://github.com/ocaml/ocaml/pull/11065

### `meio`

[meio](https://github.com/patricoferris/meio) is a user tool to observe the execution of fibers in `Eio`-based programs.

Note: `meio` requires [the custom-events PR](https://github.com/ocaml/ocaml/pull/11474) (and the sub-moduled Eio)

## Tools to debug and test for races

### `TSan`

`TSan` is a port of gcc/llvm's ThreadSanitizer for C. `TSan` can detect and report data races by installing and compiling with a special, instrumenting compiler. At runtime, TSan checks that each concurrent memory access is synchronized and reports a backtrace if it isn't.

[ screenshot ]

Note: Compiling with `TSan` instrumentation has an overhead of approximately 8x. Also `TSan` is `amd64` only for now.

### `dscheck`

`dscheck` is a library for model-checking a test under all possible interleavings. This way, you gain confidence that the tested code will work even under the most unlikely scheduling. This library has been instrumental in developing and testing `Lockfree` data structures for OCaml5.

### `Lin` and `STM`

`Lin` is a randomized testing library to help determine if an interface is sequentially consistent. `STM` is a sister library to `Lin` that furthermore lets a developer express model-based parallel tests of an interface. With these parallel testing libraries we have been able to identify and fix a number of OCaml5 issues.

### Additional resources

- https://twitter.com/sadiqj/status/1240386091569602562
- https://github.com/ocaml-bench/notes/blob/master/profiling_notes.md
- https://ocaml.org/docs/profiling
- http://ocamlverse.net/content/optimizing_performance.html
- https://github.com/ocaml/ocaml/blob/trunk/runtime/HACKING.adoc