---
tags: minutes
---

# Multicore Effects meeting notes

## 21 Jan 2022

- @talex5
- @patricoferris
- @bikal
- @sudha
- @avsm

ThomasL: porting Dream to Eio, using Lwt_eio internally. The main tutorial examples are now all working: https://github.com/aantron/dream/pull/194/files
Eio now supports OCaml 5.00, with compatibility with 4.12+domains too (as 5.00 is hard to install at the moment). We should probably make an Eio 0.1 release soon.

Patrick: collecting together all the multicore stuff here: https://github.com/patricoferris/awesome-multicore-ocaml/
Getting everything running together is tricky due to diamond dependencies, but working on it.

Bikal: has been testing with 5.00. Hit lots of problems with broken packages due to deprecated stuff being removed.

Sudha: testing 5.00 on Windows. Had some problems with CI, but now has a Windows VM for testing.

Anil: worried the Leo objects to objects. There are some arguments in the rationale document, however.

## 14 Jan 2022

- @talex5
- @patricoferris
- @avsm
- Bikal

Anil is keen for eio to track OCaml 5.00+trunk. Breaking compatibility with 4.12+domains is OK.

Patrick has an awesome-eio repository, linking to various Eio projects (link?).

Bikal is working on cohttp-eio, which is working well now.

ThomasL is looking at Dream (currently segfaults on multicore due to ssl bug). Antron is interested in Eio support. Looking for feedback on Eio multicore guide.

## 07 Jan 2022

- @talex5
- @patricoferris
- deepali
- @avsm
- Bikal
- Sudha

ThomasL added more text to the rationale document, filed issues about various missing features, addded a README example of caching, fixed some problems with error reporting, and made [Lwt_eio](https://github.com/talex5/lwt_eio), an adaptor to allow running Lwt code alongside Eio code to help with porting.

Bikal is testing Lwt_eio. Will work on creating a cohttp-eio library.

Sudha has adding polling support to the luv backend. This will allow Lwt_eio to work with that too.

Patrick has the macos backend mostly working, and it should work on ios too. Is experimenting with direct-style JSON parsing, but unclear how effects help here as it still needs to produce a lazy stream of items rather than returning one final result, as Angstrom does.

Deepali is measuring the speed of IO using eio and domainslib: [report](https://docs.google.com/document/d/11F0DJHVjdxByYJxRrhwC186S85lP0cTR-y-87_70yEA/edit) and [code](https://github.com/deepali2806/code-snippets/). Should try on tmpfs. Also, record Linux version, etc to make results reproducible.

Anil mentioned that Jake is now looking at using post-POSIX APIs for e.g. capabilities, which should work well with Eio. Martin is using Eio for a tree simulator. Bartosz is doing lock-free structures. He might want to check Eio's [promise.ml](https://github.com/ocaml-multicore/eio/blob/main/lib_eio/promise.ml) file. The other concurrency primitives could use some lock-free goodness too.

## 3rd Dec 2021

- @talex5
- @patricoferris
- deepali
- @avsm
- @ctk21
- @kayceesrk
- Bikal

Lots of discussion about how to combine domainslib and eio. It's not clear how e.g. a LIFO and a FIFO scheduler can coexist usefully in a single domain. When a task in the nested scheduler performs IO using the outer scheduler, it will block the whole inner scheduler. To fix this, the schedulers will need to be aware of each other.

For 5.00, it makes sense to provide a single library (Eio), while continuing to do research into other scheduling options. We want to avoid an Lwt/Async style split.

ThomasL has been continuing to add support for multiple domains to Eio. The Luv backend now has a lock-free queue. `make bench` has been added, which runs various multi-domain benchmarks.

Patrick is looking at adding support for GCD, based on the existing libuv backend. Is also porting jsonm to eio.

Bikal is removing the use of Lwt tasks from Tezos. Wants to make sure all CPUs can be used. Is planning to test this on Eio.

Deepali is looking at how Rust deals with having two separate threading systems (similar to our eio/domainslib split). Wants to perform a parallel grep operation on large files. Would be interesting to know the performance difference between eio and domainslib for this.

## 26th Nov 2021

- @talex5
- @patricoferris
- deepali

Short meeting as not many people available. Notes:

- Deepali is working with KC. Wants to port the Mirage libraries to use direct style. e.g. tcpip or a web server.
- talex5: just merged https://github.com/ocaml-multicore/eio/pull/99 which adds support for communicating between domains. Not optimised yet.
- Patrick: "I don’t have much time to work on effects, so what should be done about the GCD branch (it’s pretty far behind main) but to get it back up to main shouldn’t be awful if it is only filesystem support (not networking), but it would also require CI support to make sure it continues working."

## 18th June 2021

- @avsm
- @ctk21
- @kayceesrk
- @patricoferris
- @talex5

Notes:

- Eio now has a filesystem abstraction with capabilities: https://github.com/ocaml-multicore/eio#filesystem-access
Is uses `openat2` on Linux.
- Rust has a new capability system (cap-std, https://blog.sunfishcode.online/introducing-cap-std/). Might be worth checking how they do this in a cross-platform way.
- Switches need to reraise exceptions. Need to use `Printexc.raise_with_backtrace` to get correct backtraces.
- Would be useful if OCaml had chained exceptions.
This way, you can wrap an exception without losing the original (e.g. a cross-platform `Permission_denied` exception, caused by a `Unix_error` if you need the full details.)
Also useful for finalisers, to record that one exception happened while handling another.
See e.g. https://www.python.org/dev/peps/pep-3134/
- Would be good to get some early feedback on the Eio API.
- GCD on macos seems slow. Possibly due to it creating lots of extra threads because the main one is blocked.

## 11th June 2021

- @avsm
- @craigfe
- @ctk21
- @kayceesrk
- @patricoferris
- @talex5

Notes:

- Cancellation support in `uring`, and lifted to `eieio`. Can also cancel sleeps -- want to add support for e.g. `pick`.

- What should we do with this cast? (https://github.com/ocaml-multicore/ocaml-uring/blob/main/lib/uring/uring_stubs.c#L119)

- Concurrency primitives currently provided by multicore currently block the entire domain,
so these aren't immediately usable for scheduling. @kayceesrk: primitives with the right semantics
could be implemeted in terms of e.g. https://github.com/ocaml-multicore/effects-examples/blob/master/mvar/MVar.mli.

## 4th June 2021

- @avsm
- @craigfe
- @ctk21
- @kayceesrk
- @patricoferris
- @talex5

Discussion of @talex5's performance benchmarking results:


Overall takeaways:

- effects have negligible overhead for this use-case;
- `httpaf` suffering some overhead from hand-implementation of fibres.

These benchmark cases are very low-level and simple, which is not where OCaml shines against Rust. @ctk21: could we use more complex scenarios that are relatively simple to implement in OCaml, but not in lower-level languages? e.g. use of `https`, staging, application features (caching).

@kayceesrk: would like to what the impact of the GC is on these scenarios (number of minor / major GC cycles, exact memory used per request, using `v=0x800`). If the GC is having a significant effect, then we could tune it: e.g. space overhead.

Not yet benchmarking parallelism.

## 21st May 2021

- @avsm
- @ctk21
- @kayceesrk
- @talex5
- @patricoferris

The OCaml memory model paper left some things as "future work" (acquire/release). This is needed for high performance lock-free structures. If we don't add an official way to do it, people will likely try themselves and make something that only works on x86. Need to discuss this with sdolan.

Tom has a fast work-stealing system, which can be used once we need communication between domains.

Testing splice support with the eio Linux backend. It gives a very high-performance `cat` automatically. However, the fallback method of doing reads and writes via a pre-shared chunk is surprisingly slow - need to profile this.

We should test eio on the [C10k problem](https://en.wikipedia.org/wiki/C10k_problem) (and C10m).

We need a network abstraction that can handle e.g. a backend that natively supports http requests (e.g. JavaScript). Could do this by adding probing to the stdenv object.

## 14th May 2021

- @avsm
- @ctk21
- @kayceesrk
- @talex5
- sadiq

Looking at structured concurrency (https://github.com/ocaml-multicore/eio/blob/main/README.md).

- Does the `~sw` argument really need to be passed explicitly? Could it come from the context? The problem is that cleanup operations still need to run after cancellation has been requested, so doing it implicitly would be a problem.

Next steps:
- Want to try using `sendfile` as an optimisation.
- BSD netgraph in userspace?
- Try multiple domains. We probably need to get io-uring to release to runtime lock when it waits.

## 7th May 2021

- @craigfe
- @ctk21
- @kayceesrk
- @talex5

@talex5: Deployed @ewanmellor's multicore CI and enabled for `io_uring` and `eioio` :tada:

@craigfe: no updates.

@kayceesrk: gave another talk on multicore effects. Working on ParaFuzz with a student: a tool for fuzzing multicore OCaml programs with AFL. Will be submitted as a talk to the OCaml workshop.

Ideally want another multicore talk for the workshop this year. Could we submit something re. parsing with effects? Yes; we could show that a simple blocking parser can be used concurrently without modification thanks to effects, showing examples of Angstrom with and without callbacks.

Status of CTF tracing: should be merged in a few days, but doesn't include the user-level tracing yet.

## 30th April 2021

- @avsm
- @craigfe
- @kayceesrk
- @talex5

Want to allow other developers to play with modernising their async code (e.g. aantron and Dream). @talex5: we should invest some time in writing initial documentation for these libraries, even if they're going to change.

@talex5: `retro multicore` benchmarks still showing slowdown on some loads (e.g. concurrent TCP connections with sequential load) if we do parallel reads and writes on a single socket. Needs access to a baremetal machine rather than a VM to get accurate numbers (`multicore-dev`).

Potential usage of domains in Index: currently using systhreads and implementing an ad-hoc backpressure mechanism with yielding. Perhaps this could be swapped in with `lwt+effects`. `domainslib` has a parallel work queue.

@talex5: `eioio` now contains a [Sync library](https://github.com/ocaml-multicore/eioio/pull/18) with synchronisation primitives for fibres. Using a user-space CTF framework. Need CI for these multicore libraries; will require an extra runner (@talex5 to take charge of this). Can we turn on @ewanmellor's multicore CI directly for `eioio`? (We can try.)

@talex5: how to manage security of effect handlers? Transmissing an effect is insecure because all intermediate points in the stack can intercept it / potentially change its perceived environment. @kayceesrk: the latter problem is mitgated by using local / generative effect handlers (which then can't be introspected); fixing the former would require a change to remove `_` patterns for effect handlers.

@avsm: we need to support multiple domains soon too. Must ensure that libraries don't spin up domains independently or we won't use the hardware well. e.g. we have 8 cores. One library creates 8 domains for itself, and another creates 4. Instead, there needs to be some top-level scheduler that finds out what the modules require and allocates domains between them.

## 23rd April 2021

- @avsm
- @craigfe
- @ctk21
- @kayceesrk
- @talex5

@talex5: running [old httpaf benchmarks](https://github.com/ocaml-multicore/retro-httpaf-bench), comparing an implementation using Lwt_engine vs. one using `io_uring`. Would like to plug these benchmarks into some continuous system; discuss with the benchmarking team?

How do we allow a package to specify compatibility with multicore OCaml? We could have packages opt-in to a multicore switch with a dep-opt on some `base-multicore` package, or opt-out via incompatibility. @avsm to work on adding a multicore runner to OCaml-CI.

**TODO**: Setup benchmarking CI for running retro-httpaf-bench: https://github.com/ocaml-multicore/retro-httpaf-bench/. Add this to the medium-priority queue of benchmarking team.

## 16th April 2021

- @avsm
- @craigfe
- @ctk21
- @kayceesrk
- @samoht
- @talex5

@talex5: tracing works :tada: Using this to debug read / write sequentialisation in HTTP benchmarks.

@avsm: Reading TokIO, modern Rust library for IO. They're providing a separate API for `uring`.

Linked SQEs: mechanism in `uring` for expressing data-dependencies between operations (which are, by default, parallelised in the uring). @talex5: this may be necessary in httpaf, since there are sequential write pairs that should be submitted as a block.

How should fibres interact with the explicit `submit` step in uring? What API do we want to use to encode data dependencies in linked batches of requests? Discussed pros and cons of using first-class values to encode this relationship, w.r.t. error handling, avoidance of redundant syncs etc. @avsm, @kayceesrk, and @talex5 to discuss this further.

## 9th April 2021

- @avsm
- @craigfe
- @talex5
- @kayceesrk

Getting rid of explicit callbacks in `httpaf`. Have added the direct-style code to the benchmarking suite, so can now compare effects code against Lwt:

https://github.com/ocaml-multicore/retro-httpaf-bench

Optimisation work still to be done: some benchmarks are significantly improved, some significantly worst.

Been fixing up mirage-trace-viewer (last touched in 2014) to work with modern tools/libs. Can use that to do proper profiling rather than guessing where bottlenecks are.

@ctk21: still need to shape out perf tracing in a multicore world. One possible approach: using statmemprof to trace continuations from user-space. Could also try hooking into @engil's improved CTF tracing of the runtime (should be available in the next few weeks). @talex5 running CTF scheduler tracing in user-space; can we unify these? Would sequentialise GC events and user events in the same trace.

@craigfe: has a prototype with index+uring. There is a problem with the design of index which makes it hard to adapt to the uring world. A data structure depends on receiving offsets from disks, but need to update it to take a batch.

@patricoferris and @avsm working on GCD bindings, but it's a complex API. Need some more working experience from a MacOS developer; perhaps from the Swift world?

## 1st April 2021

Just @talex5 and @avsm a day earlier due to Easter break.

@talex5 looking at httpaf this week. Found a [bug in the parser](https://github.com/inhabitedtype/httpaf/pull/194). Going to continue looking at httpaf to speed it up -- an interesting insight is that the callback style makes it quite a rewrite to turn it back into direct style. Cohttp might end up being easier to port! But going to stick with httpaf for the moment.

@avsm is going to look into Tokio to see how the domain-per-uring model looks like. Suggested we

## 26th March 2021

- @avsm
- @craigfe
- @ctk21
- @kayseesrk
- @samoht
- @talex5

#### Dune internal fibres library

Talked to @jdiminio about Dune internal fibres library, which he was considering open-sourcing. Will pull out example of fibre use that is independent of the build system core. Looking to coordinate this work w/ effects-dev.

#### Multicore stack limit

Should we raise the default stack limit in multicore? @talex5: causes devs to be more defensive by allocating on the heap. @ctk21: will discuss with the wider multicore team; unclear if some limit is necessary here, and want to be careful of ruining UX.

As the stack expands, it's expanded to 2x size via copy; not segmented. Performance cost to each expansion, but should be negligible for massive stacks as logarithmically spaced.

#### Leaked captured continuations

Currently: if one doesn't call a continuation, then it will never be GCed. @kayceesrk: there's a non-trivial cost to attaching a finaliser to each one. Perhaps we could have a debug mode that does this and warns each time, similar to dynamic open-FD check that exists in OCaml. Want to enable library authors to provide fast APIs without being too defensive internally; if this requires multicore tooling then so be it. @avsm: so long as this doesn't require compromising the API.

#### Zero-copy

Should we adopt zero-copy as a goal? @talex5: good to be fast, but some concerns remain: sometimes copying is faster, sometimes passing an offset into a full buffer is a security concern (e.g. when sharing a page of memory with another process).

Currently the multicore GC doesn't move roots, since it doesn't do compaction. Could we exploit this directly in user code (scrapping bigarrays)? @kayceesrk: this has always been true, because compaction can be disabled via `OCAMLRUNPARAM`. @samoht, but it's not a conistently-available property for library developers, which is prohobitive. @ctk21: this might be a good reason for avoiding complexity (stop-the-world compaction phase). @kayceesrk: even w/ compaction we could introduce an immovable string tag.

#### Angstrom

Using effects cleans the implementation a lot. Every operation may require further input, but most inputs do not (since input is buffered); so the direct style code is more natural. ~10% speed improvement in benchmarks, but perhaps this would be amplified in a more intense benchmark (e.g. inside a GC-intensive library like Irmin).

Linux kernel version changes perf by 2x: 5.4. fast and crashy; 5.5 slow and stable. @talex5 has been bisecting between 5.4 and 5.5 to discover where the regression comes in, but difficult due to density of bugs.

#### Grand central dispatch

Progressing with @patricoferris, but is complex. `N:M:K` (kernel thread : userspace threads : domains). Each domain has two threads, a main one and a backup thread that runs stop-the-world GC when the main one is blocked. @kayceesrk / @avsm to discuss how best to meet GCD requirements from multicore end.

#### Irmin abstracted over IO

(@samoht rewrites Irmin once again...)

Parallel iteration over lists difficult to translate to direct style. Probably don't need unbounded parallelism, but some kind of parmap abstraction is needed. (Similar to the Dune fibre requirements.) @kayseesrk: ideal use-case for fibres over multiple domains w/ simple combinators; will require an extension of domainslib.

Currently using Lwt streams to receive file-system events, so need a multicore equivalent of this. Similar requirements in GCD, so will try to meet in the middle at `irmin-watcher`.

## 19th March 2021

* @kayceesrk
* @avsm
* @talex5
* @samoht
* @craigfe

talex5:
- Added promises to the interface and have unit tests
- [updated](https://github.com/talex5/httpaf/tree/eio) httpaf to the new binding code
- seems to be mostly working but Linux kernel bugs. 5.10 is a reasonable base.
- next: add support for cancellation and tracing support?
- avsm: better to get a fully effect-based version of httpaf first to see what the maximum performance is.
- fibre local storage and logs could be separate handlers. The number of hops to find out the handler will be fewer.
- backtraces is one option and the traces is the other one (lwt patches)
- kc: shagun exposes a user level api to push events into the common channel that we use for runtime tracing. CTF is not expressive enough to express multiple streams and we can. Max latency was because of a gc request and something happened during this time. The assumption that we made was that Lwt had a thread id (not obvious, but it doesnt). Couldnt demux the different threads. Chrome trace viewer separates thread/process concepts but there is support in viewers to visualise invidivual threads.
- talex5: the tracing patch to lwt added thread IDs using Lwt's thread local variables.
- kc: local state support was removed with the move to promises (there is no notion of a thread)
- avsm: can we have one buffer?
- kc: engil is integrating the ctf stuff into multicore in the next few weeks. ocaml/ocaml also does not support user effects, but multicore will soon.
- logging: discussion about how it should look and how its not polymorphic enough. That could be fixed with tagging. not clear if this needs effect handling or not.
- there's a noop operation if logs are disabled. if we use effects then we pay the costs of fibre switching.
- discussion about unifying tracing and logging but keep the compiler separate.
craigfe:
- has stubs for index but is sidetracked by the hacl* build system.

## 12th March 2021

Participants:
* @kayceesrk
* @avsm
* @ctk21
* @craigfe
* @talex5

**Goal**: Demonstrate the utility of async io against 5.0 branch.

[eioio](https://github.com/ocaml-multicore/eioio) -- effects based parallel IO for OCaml

Using modern IO APIs on each platform:

- io_uring ↔ Linux (@talex5 / @craigfe – prototype bindings used in `mirage/index`)
- GCD ↔ MacOS (@avsm / @patricoferris)
- iocp ↔ Windows

kc: are we farming out io across multiple system threads. How does this work?

e.g. you can use systhreads today for parallel IO even with single-core OCaml. We don't want unbounded parallism by default; it's too hard to program that way.

queuing is difficult and fertile research ground. what are the applications?
- lets table this and focus on io parallelism to start and come back to it.
- look at other PLs like swift etc to gain start of the art. See [Swift Concurrency Roadmap](https://forums.swift.org/t/swift-concurrency-roadmap/41611).

Use of the EIOIO scheduler requires the introduction of a `main` function (in application code) to supply effect handlers: value definitions must remain ambient.

@talex5:
- plan was to write a select-based backend
- wants to write a uring interface first and then write a new backend
- adding in new io_uring support for polling fds when ready for network access
- useful for old versions of Linux and we need backwards compat support
- has httpaf support working with it. httpaf roughly twice as fast as cohttp and then roughly twice as fast again with uring.
- avsm: use most bleeding edge kernel features and then go back to deal with compatibility
- plan is addition of promises but only when you want them. See [promises example from effects-examples](https://github.com/ocaml-multicore/effects-examples/blob/master/promises.ml).
- avsm: want to rethink all the fd stuff and what the channel abstraction is; cross-platform considerations GCD & iocp

@craigfe:
- in theory could change all the unix IO operations in Irmin Index as localised in one place, but there are a bunch of places where things have crept out.
- doing a rewrite on the Irmin Index side (with @pascutto) to make sure that larger blocks can be passed in one go
- avsm: can we use language features (?combinators?) to express the parallelism

tom: see a danger with thread pools that we want things to be super clean, but we have language parallelism that can get

We need to design in back pressure for lots of things: memory, IO, accept backlog (things we monitor with Prometheus).

We need a firehose channel for the effects work.