# The MirageOS Retreat: A Journey of Food, Cats, and Unikernels

Our journey started in Agadir, a Moroccan city right on the coast of the Atlantic sea, just south of the Atlas mountains. In Agadir, we had the best fish in the world (according to some) and amazing "cornes de gazelle," a delicious sample of Moroccan culture.

From Agadir, we went to Mirleft, a small town further south, full of square roads and beautiful reefs. That's where the MirageOS retreat took place. The venue had a kitchen and an amazing cook, a place for computers and presentations, a garden with a small pool, and a rooftop with dusty but nice views of the coast.

## About the Retreat

Both the venue and Mirleft as a whole were extremely inspiring in many ways. One of which included hacking on MirageOS, which was the main reason we came--of course, but we also enjoyed amazing food, saw old and new friends, and had a great time collaborating and creating with MirageOS.

At least once a year since the [first MirageOS retreat in 2016](https://mirage.io/blog/2016-spring-hackathon) (with a Covid break in 2020), people get together and work on anything related to MirageOS. These retreats provide a great atmosphere, working environment, and everything else that's needed to be productive and to have a wonderful time.

Besides, the retreat is always a nice opportunity to [eat our own dog food](https://en.wikipedia.org/wiki/Eating_your_own_dog_food).

The organiser, [Hannes](https://mobile.twitter.com/h4nnes) (among others), always makes sure that most of the infrastructure we rely on is running on MirageOS as much as possible. A welcome addition this year was a local [opam cache](https://hannes.robur.coop/Posts/OpamMirror), which allowed us to download and install packages without crushing the data allowance on the SIM card installed on our main access point.

## About MirageOS

[MirageOS](https://mirage.io/) is an ecosystem that constructs unikernels. In a superficial nutshell, a [unikernel](https://en.wikipedia.org/wiki/Unikernel) is a machine image that contains one process and a minimal set of operating system features the process requires. Unikernels are designed to be secure, efficient, and small. MirageOS unikernels are written in OCaml, a functional, semantically rich and type-safe programming language.

MirageOS can be used in a wide range of settings, like robust reimplementations of core system services and protocols like ([DNS](https://github.com/mirage/ocaml-dns), [SSH](https://github.com/mirage/awa-ssh), [TLS](https://github.com/mirleft/ocaml-tls), and [many more](https://github.com/mirage/)), as well as higher level applications like [web services](https://hannes.robur.coop/Posts/OpamMirror). It's also on its way to become a good choice for bare-metal applications on various chipsets, like the [Raspberry Pi 4](https://github.com/dinosaure/gilbraltar) (see also the section below on _Implementing a Jack Port Driver_)

### Our Projects & What We Learned

We worked on lots of interesting things, but let's start with the ones that directly relate to MirageOS.

#### Deploying Albatross on Nixos: No More iptables Debugging

[Albatross](https://github.com/roburio/albatross) is an orchestrator for MirageOS unikernels. It runs on a Linux system and manages unikernels using Solo5. It's made of several services, one of which is the remote TLS endpoint, which accepts requests from the network to manage the orchestrator.

I wanted to run Albatross on my favourite Linux distribution, [NixOS](https://github.com/NixOS/nixpkgs), and I hoped to be able to hack around this quickly; however, it turned out to be harder than expected. I learned so much about systemd and networking while doing this project.

A Nix flake (a new way of defining packages, which comes with many rough edges) and a NixOS module are added to the main repository in [this PR](https://github.com/roburio/albatross/pull/120).
To test that it works and to play with it, I've written a [small tutorial](https://github.com/Julow/albatross-nixos-example) that explains how to build a Qemu VM with Albatross running and how to deploy a unikernel using the remote TLS endpoint.

#### Coffee Chat Bot: a Friendly Unikernel for a Friendly Work Environment

Some of us worked on deploying a coffee chat bot as a MirageOS unikernel. Contrary to how it sounds, it isn't a robot that serves coffee (which would be extremely awesome)! Instead, it's a Slack bot that lets people on our company's Slack channel to opt-in for a coffee chat with a colleague. The coffee chat bot then matches each opt-in randomly with one other opt-in. This bot was already written in OCaml, and it's merely a single process. Due to its nature, it doesn't need to do any super complicated operating system stuff. So a natural came to us: why not make a unikernel out of it?

So [we did](https://github.com/pitag-ha/slack_bot).

Making a unikernel out of a relatively simple application sounds rather straightforward. The first step was to get rid of all Unix operations. It's incredible how many small Unix calls we were doing without even noticing. For example, we were using `Unix.time` all over the place, such as scheduling, providing a seed for the random library, and giving timestamps to our database entries.

The database posed another problem. We had been using `irmin-unix`, which writes to disk using Unix, but instead we started to use `irmin-mem`, which writes to memory. We persisted (and inspected) the data by syncing our in-memory database with a GitHub repository. If you're not familiar with Irmin, a MirageOS library, its design follows the principles of the Git design and provides a library called `irmin-git` to bridge the two.

Providing the network stack needed for the Git (and also for the Slack API) communication is one of the typical tasks the operating system needs do. In our case, that's MirageOS. MirageOS has a concept called "devices," which are the operating system features your unikernel might need. Examples of "devices" are network interfaces, network stacks, filesystems (which we didn't need), and monotonic time sources. MirageOS will provide a concrete implementation of such a device at your unikernel's compile time, as long as you declare the device in the MirageOS configuration file `config.ml`.

The things described were just a small part of our nice, educational journey making a coffee chat unikernel. One more detail that's worth mentioning: the bot now uses [`httpaf`](https://github.com/dinosaure/paf-le-chien) for the Slack API interactions. Before, it was using `cohttp`, which is already independent from Unix (unlike, for example, the OCaml `curl` wrapper `curly`). So porting it to `httpaf` wasn't technically necessary, but it was a great way to get to know and test the latest "cutting-edge" unikernel features.

#### Implementing a Jack Port Driver, or How to Make a Unikernel Sing Bare-Metal

We also went bare-metal during the retreat. "Bare-metal" sounds cool, doesn't it? Let us explain what we really mean by it. Often, the way to run a MirageOs unikernel is as follows:
- You have a Linux kernel on your machine and virtualize it via a hypervisor such as KVM.
- That hypervisor is then abstracted further by a tool called Solo5 which integrates well with MirageOs unikernels.

With this workflow, the communication between the unikernel and the hardware goes over several layers of abstraction. A "bare-metal" unikernel, on the contrary, communicates with the hardware directly, without any interfacing kernel such as Linux. The device we chose to do bare-metal work on is the Rasperry Pi 4 (RPi4).

So we needed an RPi4 bare-metal OCaml runtime. Luckily, Dinosaure wrote one last year: [Gilbraltar](https://github.com/dinosaure/gilbraltar). It also dumps the text of OCaml print statements into the UART, which is a technical way of saying that we can send such text over USB (concretely over a USB to serial TTL cable) and see it. Quite useful for debugging!

As you can see, doing bare-metal work is quite restrictive and everything that tends to be taken for granted needs to be implemented, like drivers, for example.

So that's what we decided to do.

Last year, some colleagues already implemented a driver for LED strips and powered our office's [Christmas tree](https://twitter.com/Dinoosaure/status/1471128595154231300) with a bare-metal OCaml RPi4! What is cooler than [making our bare-metal RPi4 Christmas tree sing](https://tarides.com/blog/2021-11-11-mirageos-workshop-working-with-the-raspberry-pi-4)? Well, a lot of things are (and we love music), so we decided to implement a jack port driver.

Jack port drivers on a digital device are an interesting concept. Digital devices are digital, but jack ports expect analog data. One way the RPi4 can handle that is via a concept called PWM: Pulse Width Modulation. The PWM modulates analog signals (i.e., values between 0 and 1) by sending digital signals (i.e., either 0 or 1) really fast.

That modulation is done on the hardware side of the RPi4, concretely on a RPi4 _peripheral_ also called PWM. [Peripherals](https://datasheets.raspberrypi.com/bcm2711/bcm2711-peripherals.pdf) are RPi4 hardware devices that are mapped to specific address ranges in the RPi4's memory. You communicate with them by writing to or reading from those locations in memory. The address range of each peripheral is structured into registers. One example of a register of the PWM is the PWM FIFO, i.e., the hardware queue that stores the data flowing from the program to the jack port.

[Our jack port driver](<TODO:insert link>) does two things--both by writing to and reading from the right places in the PWM memory range.
1. It can initiate the RPi4 for jack port communication (e.g., it sets the RPi4's clock to the correct frequency at which the port reads data from the FIFO)
2. It configures the correct modes to ensure the right data flow and that it can actually write data to the FIFO--without overflowing it.

We convert the music into the right binary format to [make the RPi4 "sing" bare-metal](<TODO:insert link>) by simply using `ffmpeg`. Then we write a program with that music in memory using the MirageOS tool [`ocaml-crunch`](https://github.com/mirage/ocaml-crunch). That program just calls the driver to do the rest and is compiled for the RPi4 target with `gilbraltar`.

This work is stronly related to MirageOs in three ways. First, the program playing music bare-metal on the RPi4 is a unikernel written in OCaml. It's compiled to a RPi4 disk image containing one process playing music, and the minimal set of operating system features needed for that: UART support (for debugging) and a jack port driver. Second, the program is compiled with `gilbraltar`, which forms part of the MirageOS ecosystem and whose design and implementation is based on core tools in the MirageOs ecosystem, such as Solo5 and `ocaml-solo5`. Third, we added a layer of abstraction to the jack port driver and made it a MirageOS "device," so one could use the driver while also leveraging other MirageOS features that work bare-metal on a RPi4.

#### Monitoring mirage.io and Chasing Memory Leaks

One of the MirageOS goals is to be able self host our infrastructure. At the retreat, many tools we used were based on the MirageOS ecosytem: a DNS resolver ([mirage/ocaml-dns](https://github.com/mirage/ocaml-dns)), an opam repository cache ([robur/opam-mirror](https://git.robur.io/robur/opam-mirror)), and a portable file transfer application ([dinosaure/bob](https://github.com/dinosaure/bob)). It's not a surprise that the official website, [mirage.io](https://mirage.io), is a unikernel itself. However, in the past six months, we experienced two website crashes due to `Out_of_memory` exceptions. The unikernel is configured to run with 1GB of RAM, so that's a slow running memory leak that requires investigation.

The question is how to investigate such a leak ?

##### Locally?

The initial attempt consisted in _tracing_ memory allocations using `statmemprof` while bombarding the server with requests by using benchmarking tools such as ApacheBench (`ab`) or `wrk`. `statmemprof` is an implementation of _Statistical Memory Profiling_ in the OCaml runtime. It enables sampling allocations at a fixed rate and tracing values until they are garbage collected. Using [memtrace-viewer](https://github.com/janestreet/memtrace_viewer), one can analyse the memory usage and see which values are still live when the program goes out of memory, for example. For a unikernel with network access, it's possible to add an endpoint to enable tracing on demand: [roburio/memtrace-mirage](https://github.com/roburio/memtrace-mirage).

Unfortunately, this setup didn't help us identify the leak. Indeed, we can still expect the server to work fine under normal conditions. Somehow we need to understand which rare event, leaking a small bit of memory at a time, is happening enough times to consume all available memory.

##### Monitoring the Live Unikernel

Only the _Real™_ Internet would tell us the answer, so we monitored the live unikernel application. [roburio/mirage-monitoring](https://github.com/roburio/mirage-monitoring) was of great help, as it enables two things:
- Reporting application-wide metrics to an InfluxDB endpoint, which can be displayed using Grafana
- Changing logs level/metrics sources at runtime

Adding `mirage-monitoring` to a unikernel was surprisingly easy. It was only a matter of updating the configuration file with some functoria voodoo: https://github.com/mirage/mirage-www/pull/767. At some point, it will upstreamed in the `mirage` tool so that adding monitoring is a single-line job. The hard part was providing the unikernel two network stacks to expose one to the internet while keeping the other for internal use only.

Next, we set up a typical Grafana deployment using InfluxDB/Telegraf for the metrics input and data storage. Logs were displayed using `albatross-client-local`.

##### Chasing the Leak

Now we can see the numbers for the live website. Memory usage, indeed, but also other metrics were included by default, such as the number of established connections in the TCP stack. There we found the source of the leak. Throughout the day, the number of established TCP connections kept increasing.

Finally at runtime, we temporarily changed the TCP stack's log level to _debug_, monitor the logs, and wait for the moment where the number of established TCP connections would increase without decreasing afterwards. These logs described what was going on in the TCP stack at the exact moment the connection leak happened. At this point, we figured out that it occurred when a client connected to the server but fail to perform the TLS handshake, so the server dropped the connection without _closing_ it--hence leaking it _forever_.

Here we go: [_one less leak_](https://github.com/dinosaure/paf-le-chien/pull/72).

##### Next Steps

Matching _logs_ and _metrics_ to inspect them together has proven to be very useful. We used Grafana for metrics, so the next step would be to also provide logs because Grafana supports structured logging through the [Loki](https://grafana.com/oss/loki/) logs aggregation system.

#### MirageHole - A Unikernel DNS Resolver with Holes

One way to stop web trackers, advertisements, and malware is to block access to sites known to contain such things. A popular approach is through browser extensions like [AdBlock](https://en.wikipedia.org/wiki/AdBlock) and [Privacy Badger](https://en.wikipedia.org/wiki/Privacy_Badger). Another approach known as [a DNS sinkhole](https://en.wikipedia.org/wiki/DNS_sinkhole) is installing a local DNS server that resolves bad domains to an invalid IP address. This approach has the advantage of working across different operating systems, browsers, and devices (laptops, smartphones, smart-TVs, etc.). For an added bonus, it can also save network bandwidth.

Another project initiated during this year's retreat was to implement [Mirage-hole](https://github.com/jmid/mirage-hole): a DNS sinkhole running as a Mirage Unikernel. It was inspired by [Pi-hole](https://en.wikipedia.org/wiki/Pi-hole) for the Raspberry Pi. Starting from a DNS-stub example from [dnsvizor](https://github.com/roburio/dnsvizor) (and after a bit of network debugging), we got a unikernel running that would block a single selected domain. We then extended this to fetch and parse [a blocklist](https://github.com/blocklistproject/Lists) at start-up. Next, we integrated a little webserver to serve statistics about the requested and blocked domains. Overall, the project was a nice opportunity to talk to and learn from several MirageOS contributors, and it served as a nice tour-de-force of several MirageOS networking libraries.

#### Tarides Map - Serving the Tarides Geographical Distribution in a Unikernel

Tarides Map is a project intended to show the geographic distribution of all Tarides collaborators as a website. At the retreat, we explored deploying the site in a unikernel. To do this, we had to decide how to serve the files on the server and integrate it into a unikernel. We had two options use `ocaml-crunch` or Docteur.

We initially used [Docteur](https://github.com/dinosaure/docteur) due to an inspiration from a different project called [Pasteur](https://github.com/dinosaure/pasteur), which uses Docteur and is deployed in a unikernel as a static site, which was exactly what we were aiming to do with Tarides Map. However, integrating Docteur into the project proved to be more difficult than we had expected. One reason was that Solo5 isn't currently supported on MacOS, the operating system used to write the project at the retreat. After compiling to Unix instead and numerous hours debugging, we were eventually able to generate the disk image; however, we still had issues deploying it in a unikernel, so we decided to try using [ocaml-crunch](https://github.com/mirage/ocaml-crunch) instead.

Using `ocaml-crunch` proved to be a more straightforward option. We merely had to move some files around so that the directory structure could be turned into a standalone OCaml module to serve the file contents without requiring an external filesystem to be present. After doing this, we were successfully able to deploy the site [here](https://github.com/SaySayo/tarides_map_static_website).

### What We Dreamed About

Another very interesting part of the retreat were the *dreaming sessions* organized by Hannes. The central idea behind this exercice was to allow ourselves to dream about how we envision the MirageOS project in the future, no matter how untangible and seemingly unrealistic. We talked about those dreams in two sessions.

The initial session revolved around gathering these dreams and ideas, without discussing how to achieve them, and let our mind go free with what we wanted to accpomplish with MirageOS. Often times, those dreams would be shared with other participants. Some dreamed about replacing their whole software infrastructure by MirageOS, if not their main operating system! Others dreamed of artistic applications for Mirage, like using it as a backbone for musical endeavors.

The subsequent session revolved around *how* we could reach those dreams. This facilitated a more practical discussion around the challenges we may face along the way. Interestingly enough, in some instances, it turned out some dreams were either already achieved (like reverse-debugging Solo5!) or were close to being achievable.

A beautiful example of the attendees' dedication is that it did not take long for some to start working on projects like MirageOS-OS, a hypervisor for MirageOS unikernels and written with MirageOS, or to successfully implement a jack port driver for the Raspberry Pi 4, bringing us closer to MirageOS powered synthesisers and to MirageOs midi interfaces!

### What Else We Worked On and Learned

As mentioned above, the retreat was extremely inspiring, even with respect to topics less related to MirageOS than the ones mentioned here. The one we're most proud of is our Mirleft MirageOS EP that contains five tracks (_five_ in the spirit of Solo*5* and OCaml *5.0*, of course). Its genre might be better described as *OCamlwave*! On our EP, you will find many musical oddities ranging from an on-premise recorded drum solo (with glasses, cloth-racks, and flip-flops) to a cat-powered cover of *Mr Sandman* (as an hommage to our time singing to Morroco's many, *many* cute cats.) to the occasional dramatic rendition of controversial pull requests on the OCaml compiler.

However, not everything in Mirleft was about music and animals. Some things were also about the beauitful waves.

Mirleft is a paradise for surfing, both for beginners and advanced surfers! We went to a nice sandy beach with perfect conditions to get started with surfing. Advanced surfers would probably go to one of the reefs for surfing, which we, in turn, found amazing for a peaceful walk, sometimes with and sometimes without company from a street dog we christened `null`.

And, well, talking about `null` (apart from naming street animals), we also had plenty of other computer science related conversations at the retreat. All of them were extremely enriching! A couple of examples include exception backtracing in LWT programs, and BGP intrinsics.

## Thanks for All the Fish!

As this lengthy report can attest, our experience was an amazing one for all. The MirageOS Hack Retreats are always an otherworldly space, where amazing individuals gather to exchange thoughts and create new (and better) software. Friends are made along the way, some bugs are fixed, new ones are found, and great new ideas emerges.

This very special sense of community is rare, so we would like to thank everyone who organized, attended, and tended to the event. Thank you to our delightful hosts, who've been with us since the first retreat in 2016! Thank you as well to [Hannes and Robur](https://robur.io) for organizing those retreats and spending time instilling the same inspiration in the great project that is MirageOS! Finally, thank you to old and new friends, as well as old and new MirageOS hackers, for this amazing week of happy banter and hacking!