---
# System prepended metadata

title: OCaml Platform Installation Spec (Draft)
tags: [drafts]

---

---
tags: drafts
---

# OCaml Platform Installation Spec (Draft)

As mentioned in previous RFCs, we identified painpoints related to the OCaml Platform and they can be summurized as follow:

- The installation takes too much time (both for the initial setup and the creation of new sandboxes)
- The Platfom usage is not unified (fragmentation into ~10 tools)
- OCaml Platform setup is a tedious task for both newcomers and experience users

Previous RFCs have tried to propose solutions that would address all of these problems, but it has proven to be too large of a scope. In this spec, we propose a solution that focuses solely on the last item: easing the Platform installation.

We recently released a version of the VSCode extension that was intended to address the following problems:

- The Platform tools conflict with the users projects dependencies
- The Platform tools have to be installed manually for every switch
- Some Platform tools depend on the project configuration (e.g. ocamlformat)

We tried to address these by automating the installation of the Platform tools in a separate opam root.

However, user feedback showed some drawbacks in our approach, in particular, the lack of flexibility for experienced users. In this document, we propose a new solution that takes this feedback into account.

## Workflow

Detection mechanism when first starting:
- Is there an opam binary that is a suitable version?


```flow
st=>start: Start
e=>end: End

opam_detected=>condition: Opam binary
detected?

install_opam=>operation: Install opam
init_opam=>operation: Init opam

st->opam_detected
opam_detected(yes)->e
opam_detected(no)->install_opam->init_opam

opam_detected->opam_installed

```

### Installation

#### Compiler Selection

Which version of OCaml do we initialise if one isn't selected? We need the latest version that supports lsp-server.  For example [4.12 had a lag](https://github.com/ocaml/opam-repository/issues/17530) and no lsp-server.



#### Tool Selection

The list of OCaml Platform tools is well detailed [here](https://ocaml.org/platform/), but to be specific, we are considering the following subset to be the typical tools installed in a development environment:

- Opam - Package manager
- Dune - Build system
- Utop - OCaml toplevel
- Merlin - Syntax completion
- Odoc - Documentation generator
- Mdx - Markdown code block execution
- LSP - OCaml's Language server protocol
- OCamlformat - OCaml formatter
- Dune-release - CLI to release packages on Opam

We are intentionnally ommitting Platform tools such as `ppxlib` as they are to be installed as a library with Opam, and are less impacted by the problems we are discussing here.

From this list, we classify the tools into different categories

|              | Can be outside of switch | Doesn't depend on project | Doesn't depend on compiler | Can install from binaries |
| ------------ | ------------------------ | ------------------------- | -------------------------- | ------------------------- |
| Opam         | ✅                        | ✅                         | ✅                          | ✅                         |
| Dune         | ✅                        | ❌ [^1]                    | ✅                          | ❌                         |
| Utop         | ✅                        | ✅                         | ✅                          | ❌                         |
| Merlin       | ✅                        | ✅                         | ❌                          | ❌                         |
| Odoc         | ✅                        | ✅                         | ❌                          | ❌                         |
| Mdx          | ✅                        | ✅                         | ✅                          | ❌                         |
| LSP          | ✅                        | ✅                         | ❌                          | ❌                         |
| OCamlformat  | ✅                        | ❌                         | ❌ [^2]                     | ❌                         |
| Dune-release | ✅                        | ✅                         | ✅                          | ❌                         |

[^1]: Dune depends on the user's project but is backward compatible, so it can be considered project-independent if using the latest version

[^2]: OCamlformat can be compiled with the latest version of the compiler

Questions:

- Can we use a precompiled utop instead of compiling it with Dune?
    - A: Yes, and this is now the recommended way of doing. However, this impacts REPL integration in VSCode and we'll have to update it to load the project libraries manually for the user.
- Are we sure we can use Dune outside of the user's sandbox? How does it impact RPC?
    - A: ???

### Possible Directory Layout

```
.opam/tools/
         / dune
            / 2.8 
              / dune
         / opam
            / 2.0.7
              / opam
            / 2.1.0
              / opam
.opam/4.10/
         / odoc
            / 2.0.0
              / odoc
         / lsp-server
            / 1.8.0
              / ocaml-lsp-server
         / ocamlformat
           /0.15.0
              / ocamlformat
           /0.14.0
              / ocamlformat
_opam/bin
   symlinks to above ^^^

```

With this classification, we can decide how to install each specific tools:

- **Opam**: Download the latest binary, keep in a global store
- **Dune**: Build the latest version in a sandbox with the latest compiler and save in a global store
- **Utop**: Install from sources in the sandbox
- **Merlin**: Detect the sandbox compiler, build in a sanbox with the same compiler and save in a global store
- **Odoc**: Detect the sandbox compiler, build in a sanbox with the same compiler and save in a global store
- **Mdx**: Build in a sandbox with the latest compiler and save in a global store
- **LSP**: Detect the sandbox compiler, build in a sanbox with the same compiler and save in a global store
- **OCamlformat**: Detect the project's version, detect the compiler version, install the correct version in a sandbox with the same compiler and save in a global store
- **Dune-release**: Build in a sandbox with the latest compiler and save in a global store

The Platform tools are intended to be used as binaries, and users should not need their sources (with the exception of `utop` possibly[^3]). So we can cache the compiled binaries and re-use them. If the correct version of the binary can be found, it is copied to the user's opam switch. If not, they are compiled or downloaded and copied thereafter.

[^3]: The user requires the sources of Utop because dune compiles it when running `dune utop`. Not sure if that's something we want to support? The REPL integration in VSCode could load the libraries itself.

Note that the installation should be idempotent: if the current switch is already setup, we don't need to do anything.
This property is important for the tools executation (see below).

### Upgrade

There are broadly two kinds of update:

- Updates of tools installed with pre-compiled binaries
- Updates of tools installed from Opam

For the former, we will hardcode the versions in the code (because we need to verify the checksums and eventually the signature of the binary). If the version in the code is not the version of the latest installed binary, we download it and copy it in the binary store.

For the later, we first update the opam-repository of the opam switch used for the Platform tools compilation.
Once synched, we detect the project's versions and compare it with the latest versions in the opam-repository. For the set of tools that have a new version in the opam repository, we run the installation flow with the latest version.

### Execution

If we keep the binaries outside of the switch:

- How do we allow the user to select which version to use?

If we copy the binaries in the switch:

- How do we follow up with the project update? (e.g. update dune version, update ocamlformat)

Solution: we run make sure the tools are installed every time we execute them.

This worflow will be used by VSCode, but let's illustrate it with some command lines:

```sh
ocaml-platform opam # Uses the latest opam outside of the user's switch
ocaml-platform utop # Uses the utop installed in the switch
ocaml-platform merlin # Uses the merlin binary copied in the switch
ocaml-platform ocamlformat # Checks the project's config and install the correct version
ocaml-platform dune # Uses the merlin binary copied in the switch
```

An execution of a tool will trigger an installation confirmation if it is not installed, so this fits perfectly with VSCode's workflow.

#### Alternative with opam plugin


```
opam-version: "2.0"
depends: [
  "ocaml" {>= "4.04.0"}
  "dune" {build & >= "1.2.0"}
  "ounit" {with-test & >= "1.0.2"}
  "ppx_sexp_conv" {with-test & >= "v0.9.0"}
  "re" {>= "1.9.0"}
  "stringext" {>= "1.4.0"}
  "odoc" {with-doc & >= "1.5.0"}
]
x-dev-tools: [
  "ocamlformat" {="0.15.0"}
  "odoc" {>="2.0.0"}
  "ocaml-lsp-server" {<"1.5"}
]
build: [
  ["dune" "subst"] {pinned}
  ["dune" "build" "-p" name "-j" jobs]
  ["dune" "runtest" "-p" name "-j" jobs] {with-test}
]
```

A possible version for the future:

```
opam-version: "3.0"
depends: [
  "ocaml-stdlib" {>= "4.04.0"}
  "ounit" {with-test & >= "1.0.2"}
  "re" {>= "1.9.0"}
  "stringext" {>= "1.4.0"}
]
tools: [
  "ocaml" {=ocaml-stdlib:version}
  "dune" {with-build & >= "1.2.0"}
  "ocamlformat" {="0.15.0"}
  "odoc" {with-doc & >="2.0.0"}
  "ocaml-lsp-server" {with-dev & <"1.5"}
  "ppx_sexp_conv" {with-test & >= "v0.9.0"}
]
build: [
  ["dune" "subst"] {pinned}
  ["dune" "build" "-p" name "-j" jobs]
  ["dune" "runtest" "-p" name "-j" jobs] {with-test}
]
```



x-dev-tools is for tools to operate on your code:
- for binary depends only
- packages not added to OCAMLPATH (so no OCaml libraries are exposed to local switch)
- can be compiled against a specific OCaml version
- does not constrain the main project (e.g. for ocaml-ci) for the tool depends
- 
What about bisect_ppx? Both a library and a tool.
Future feature: add support for dune workspaces into VSCode extension.

## Usage

Here we list the different use cases per user.


### In VSCode

- Install the platform tools if not present in the sandbox
    - Install opam first
    - Copy in sandbox or 
- Visualize the Platform tools with their versions in a treeview
- Select the version of a tool to use
- Run a tool with given arguments

### In CI/CD

- Integration tests
- Use for `setup-ocaml`?

### Final user

- Install a platform tool manually
    - Using e.g. `opam install ocamlformat`
    - Should override the binary if already installed


## Output

The output of the project is a library that encodes the above workflow and use cases with the following properties:

- Is IO-agnostic so that it can be compiled to JS and be used in VSCode
- Generic interface for notifications, confirmations and progress

In addition, a CLI that replicates VSCode's features for integration test purposes:

- `ocaml-platform status` - Describe the state of the Platform tools (version, installed / outdated, etc.)
- `ocaml-platform setup` - Install OCaml Platform
- `ocaml-platform <binary>` - Redirect the arguments to the correct binary

We will provide a NodeJS implementation of this library for use in VSCode and a Unix implementation for use in CI/CD.

## QA

Summary of the above spec in a Question-Answer format

**Where do we store the binaries?**

The binaries are stored in a directory with user access where the path encode information needed for the selection of the binary.

Typically:

`~/.cache/ocaml/bin/ocamlformat/ocaml-4.18.0/0.17.0/ocamlformat`

We reference this path `~/.cache/ocaml/bin/` as the "Binary store".

**How do we install Opam?**

Opam binary is downloaded from Github and stored in the binary store as we might not have access to `/usr/bin`.

If an Opam installation is already present, we still copy Opam binary in the binary store so we have total control over the environment.

**How do we install the platform tools?**

Each platform tools has a specific installation flow. But generally, we will:

- Run `opam install` in the user switch if the sources are required
- Download the binary if available and save it in the binary store
- Build the binary from sources in a switch with the correct OCaml version if binaries are not available, then save it in the binary store

Once the binaries are in the binary store, they are copied in the user project.


## User stories

TODO: Write user stories for all of the user workflows we want to support:

- Bucklescript users want to use the tools
- Esy users don't want to install opam
- User want to use a specific version of a tool
- User want to pin a tool
- User needs the sources of a tool
- User want to use a custom opam repository

Notes on user archetypes: https://docs.google.com/document/d/1tk3VDPja-v25aiNoIXqFSlyjBmE3db1f9k2auq3BcMo/edit?usp=sharing
