# Statmemprof Resurrected
Nick Barnes, Tarides, 2024-01-16
<style>
.reveal {
font-size: 36px;
}
</style>
---
## "Statmemprof": Statistical Memory Profiler
- Sample all allocations at some frequency ("lambda") per word.
- Callbacks into Caml at allocation, promotion, and de-allocation.
- Allocation callbacks optionally take stack backtrace.
- Very low overhead at useful values of lambda.
- Jacques-Henri Jourdan, Stephen Dolan; 2019/2020.
- Used in various tools, notably <font color="green">`memtrace`</font>.
- Removed for multicore.
---
## Original "experimental" API
```ocaml
Gc.Memprof:
val start : sampling_rate:float ->
?callstack_size:int ->
('minor, 'major) tracker ->
unit
val stop : unit -> unit
(* stops sampling and callbacks *)
```
---
## Original implementation
- Use existing allocation check;
- No recompilation, unsampled allocations unchanged;
- Defer callbacks to safe points;
- Sampling off during callbacks;
- Tricky interface to systhreads.
---
## Multicore questions
- Per-domain ?
- Multiple profiles ?
- Memprof races between different domains ?
- <font color="green">`Domain.spawn`</font> ?
- Which domains run callbacks ?
- Domain termination ?
---
## Multicore answers
- Per-domain ? *Yes*
- Multiple profiles ? *Yes*
- Memprof races between different domains ? *Atomics, and exceptions*
- <font color="green">`Domain.spawn`</font> ? *Inherit*
- Which domains run callbacks ? *Original domain if possible*
- Domain termination ? *Other domains inherit state and callbacks*
Introduction of multiple profiles, and domain termination, motivates a new commitment to run callbacks.
---
## New API
```ocaml
Gc.Memprof:
type t
val start : sampling_rate:float ->
?callstack_size:int ->
('minor, 'major) tracker ->
t
val stop : unit -> unit
(* stops sampling *)
val discard : t -> unit
(* stops callbacks *)
```
No change to <font color="green">`tracker`</font> type.
---
## New implementation
- Much from scratch;
- More abstraction; clean interfaces to systhreads, domains, GC;
- <font color="green">`Gc.Memprof.t`</font> is a block on the Caml heap.
---
## Future API???
```ocaml
val join : t -> unit
val current : unit -> t option
val leave : unit -> unit
val suspended : ('a -> 'b) -> 'a -> 'b
val stats : t -> stats (* ??? *)
(* ... ??? *)
```
---
## Current status
- <font color="green">`nickbarnes/ocaml`</font> fork, <font color="green">`nick-11911-statmemprof-rebase`</font> branch;
- Passes <font color="green">`statmemprof`</font> tests, and additional tests;
- Provokes some failures in multicore tests;
- Rebasing now to prepare for PR.
{"title":"Statmemprof resurrected","description":"Statmemprof ResurrectedNick Barnes, Tarides, 2024-01-16","contributors":"[{\"id\":\"538331cc-feed-422c-9d17-2e90cc33e464\",\"add\":2906,\"del\":81}]"}