OCaml Platform Installation Spec (Draft)

As mentioned in previous RFCs, we identified painpoints related to the OCaml Platform and they can be summurized as follow:

The installation takes too much time (both for the initial setup and the creation of new sandboxes)
The Platfom usage is not unified (fragmentation into ~10 tools)
OCaml Platform setup is a tedious task for both newcomers and experience users

Previous RFCs have tried to propose solutions that would address all of these problems, but it has proven to be too large of a scope. In this spec, we propose a solution that focuses solely on the last item: easing the Platform installation.

We recently released a version of the VSCode extension that was intended to address the following problems:

The Platform tools conflict with the users projects dependencies
The Platform tools have to be installed manually for every switch
Some Platform tools depend on the project configuration (e.g. ocamlformat)

We tried to address these by automating the installation of the Platform tools in a separate opam root.

However, user feedback showed some drawbacks in our approach, in particular, the lack of flexibility for experienced users. In this document, we propose a new solution that takes this feedback into account.

Workflow

Detection mechanism when first starting:

Is there an opam binary that is a suitable version?

Installation

Compiler Selection

Which version of OCaml do we initialise if one isn't selected? We need the latest version that supports lsp-server. For example 4.12 had a lag and no lsp-server.

Tool Selection

The list of OCaml Platform tools is well detailed here, but to be specific, we are considering the following subset to be the typical tools installed in a development environment:

Opam - Package manager
Dune - Build system
Utop - OCaml toplevel
Merlin - Syntax completion
Odoc - Documentation generator
Mdx - Markdown code block execution
LSP - OCaml's Language server protocol
OCamlformat - OCaml formatter
Dune-release - CLI to release packages on Opam

We are intentionnally ommitting Platform tools such as ppxlib as they are to be installed as a library with Opam, and are less impacted by the problems we are discussing here.

From this list, we classify the tools into different categories

	Can be outside of switch	Doesn't depend on project	Doesn't depend on compiler	Can install from binaries
Opam	✅	✅	✅	✅
Dune	✅	❌ ^[1]	✅	❌
Utop	✅	✅	✅	❌
Merlin	✅	✅	❌	❌
Odoc	✅	✅	❌	❌
Mdx	✅	✅	✅	❌
LSP	✅	✅	❌	❌
OCamlformat	✅	❌	❌ ^[2]	❌
Dune-release	✅	✅	✅	❌

Questions:

Can we use a precompiled utop instead of compiling it with Dune?
- A: Yes, and this is now the recommended way of doing. However, this impacts REPL integration in VSCode and we'll have to update it to load the project libraries manually for the user.
Are we sure we can use Dune outside of the user's sandbox? How does it impact RPC?
- A: ???

Possible Directory Layout

.opam/tools/
         / dune
            / 2.8 
              / dune
         / opam
            / 2.0.7
              / opam
            / 2.1.0
              / opam
.opam/4.10/
         / odoc
            / 2.0.0
              / odoc
         / lsp-server
            / 1.8.0
              / ocaml-lsp-server
         / ocamlformat
           /0.15.0
              / ocamlformat
           /0.14.0
              / ocamlformat
_opam/bin
   symlinks to above ^^^

With this classification, we can decide how to install each specific tools:

Opam: Download the latest binary, keep in a global store
Dune: Build the latest version in a sandbox with the latest compiler and save in a global store
Utop: Install from sources in the sandbox
Merlin: Detect the sandbox compiler, build in a sanbox with the same compiler and save in a global store
Odoc: Detect the sandbox compiler, build in a sanbox with the same compiler and save in a global store
Mdx: Build in a sandbox with the latest compiler and save in a global store
LSP: Detect the sandbox compiler, build in a sanbox with the same compiler and save in a global store
OCamlformat: Detect the project's version, detect the compiler version, install the correct version in a sandbox with the same compiler and save in a global store
Dune-release: Build in a sandbox with the latest compiler and save in a global store

The Platform tools are intended to be used as binaries, and users should not need their sources (with the exception of utop possibly^[3]). So we can cache the compiled binaries and re-use them. If the correct version of the binary can be found, it is copied to the user's opam switch. If not, they are compiled or downloaded and copied thereafter.

Note that the installation should be idempotent: if the current switch is already setup, we don't need to do anything.
This property is important for the tools executation (see below).

Upgrade

There are broadly two kinds of update:

Updates of tools installed with pre-compiled binaries
Updates of tools installed from Opam

For the former, we will hardcode the versions in the code (because we need to verify the checksums and eventually the signature of the binary). If the version in the code is not the version of the latest installed binary, we download it and copy it in the binary store.

For the later, we first update the opam-repository of the opam switch used for the Platform tools compilation.
Once synched, we detect the project's versions and compare it with the latest versions in the opam-repository. For the set of tools that have a new version in the opam repository, we run the installation flow with the latest version.

Execution

If we keep the binaries outside of the switch:

How do we allow the user to select which version to use?

If we copy the binaries in the switch:

How do we follow up with the project update? (e.g. update dune version, update ocamlformat)

Solution: we run make sure the tools are installed every time we execute them.

This worflow will be used by VSCode, but let's illustrate it with some command lines:

ocaml-platform opam # Uses the latest opam outside of the user's switch
ocaml-platform utop # Uses the utop installed in the switch
ocaml-platform merlin # Uses the merlin binary copied in the switch
ocaml-platform ocamlformat # Checks the project's config and install the correct version
ocaml-platform dune # Uses the merlin binary copied in the switch

An execution of a tool will trigger an installation confirmation if it is not installed, so this fits perfectly with VSCode's workflow.

Alternative with opam plugin

opam-version: "2.0"
depends: [
  "ocaml" {>= "4.04.0"}
  "dune" {build & >= "1.2.0"}
  "ounit" {with-test & >= "1.0.2"}
  "ppx_sexp_conv" {with-test & >= "v0.9.0"}
  "re" {>= "1.9.0"}
  "stringext" {>= "1.4.0"}
  "odoc" {with-doc & >= "1.5.0"}
]
x-dev-tools: [
  "ocamlformat" {="0.15.0"}
  "odoc" {>="2.0.0"}
  "ocaml-lsp-server" {<"1.5"}
]
build: [
  ["dune" "subst"] {pinned}
  ["dune" "build" "-p" name "-j" jobs]
  ["dune" "runtest" "-p" name "-j" jobs] {with-test}
]

A possible version for the future:

opam-version: "3.0"
depends: [
  "ocaml-stdlib" {>= "4.04.0"}
  "ounit" {with-test & >= "1.0.2"}
  "re" {>= "1.9.0"}
  "stringext" {>= "1.4.0"}
]
tools: [
  "ocaml" {=ocaml-stdlib:version}
  "dune" {with-build & >= "1.2.0"}
  "ocamlformat" {="0.15.0"}
  "odoc" {with-doc & >="2.0.0"}
  "ocaml-lsp-server" {with-dev & <"1.5"}
  "ppx_sexp_conv" {with-test & >= "v0.9.0"}
]
build: [
  ["dune" "subst"] {pinned}
  ["dune" "build" "-p" name "-j" jobs]
  ["dune" "runtest" "-p" name "-j" jobs] {with-test}
]

x-dev-tools is for tools to operate on your code:

for binary depends only
packages not added to OCAMLPATH (so no OCaml libraries are exposed to local switch)
can be compiled against a specific OCaml version
does not constrain the main project (e.g. for ocaml-ci) for the tool depends

What about bisect_ppx? Both a library and a tool.
Future feature: add support for dune workspaces into VSCode extension.

Usage

Here we list the different use cases per user.

In VSCode

Install the platform tools if not present in the sandbox
- Install opam first
- Copy in sandbox or
Visualize the Platform tools with their versions in a treeview
Select the version of a tool to use
Run a tool with given arguments

In CI/CD

Integration tests
Use for setup-ocaml?

Final user

Install a platform tool manually
- Using e.g. opam install ocamlformat
- Should override the binary if already installed

Output

The output of the project is a library that encodes the above workflow and use cases with the following properties:

Is IO-agnostic so that it can be compiled to JS and be used in VSCode
Generic interface for notifications, confirmations and progress

In addition, a CLI that replicates VSCode's features for integration test purposes:

ocaml-platform status - Describe the state of the Platform tools (version, installed / outdated, etc.)
ocaml-platform setup - Install OCaml Platform
ocaml-platform <binary> - Redirect the arguments to the correct binary

We will provide a NodeJS implementation of this library for use in VSCode and a Unix implementation for use in CI/CD.

QA

Summary of the above spec in a Question-Answer format

Where do we store the binaries?

The binaries are stored in a directory with user access where the path encode information needed for the selection of the binary.

Typically:

~/.cache/ocaml/bin/ocamlformat/ocaml-4.18.0/0.17.0/ocamlformat

We reference this path ~/.cache/ocaml/bin/ as the "Binary store".

How do we install Opam?

Opam binary is downloaded from Github and stored in the binary store as we might not have access to /usr/bin.

If an Opam installation is already present, we still copy Opam binary in the binary store so we have total control over the environment.

How do we install the platform tools?

Each platform tools has a specific installation flow. But generally, we will:

Run opam install in the user switch if the sources are required
Download the binary if available and save it in the binary store
Build the binary from sources in a switch with the correct OCaml version if binaries are not available, then save it in the binary store

Once the binaries are in the binary store, they are copied in the user project.

User stories

TODO: Write user stories for all of the user workflows we want to support:

Bucklescript users want to use the tools
Esy users don't want to install opam
User want to use a specific version of a tool
User want to pin a tool
User needs the sources of a tool
User want to use a custom opam repository

Notes on user archetypes: https://docs.google.com/document/d/1tk3VDPja-v25aiNoIXqFSlyjBmE3db1f9k2auq3BcMo/edit?usp=sharing

Dune depends on the user's project but is backward compatible, so it can be considered project-independent if using the latest version ↩︎
OCamlformat can be compiled with the latest version of the compiler ↩︎
The user requires the sources of Utop because dune compiles it when running dune utop. Not sure if that's something we want to support? The REPL integration in VSCode could load the libraries itself. ↩︎