WIP INTERNAL TO OCAML PLATFORM TEAM - DO NOT SHARE

Welcome to the eighth edition of the OCaml Platform newsletter!

In this November+December edition, we are excited to bring you the latest on the OCaml Platform, continuing our tradition of highlighting recent developments as seen in previous editions. To understand the direction we're headed, especially regarding development workflows and user experience improvements, check out our roadmap.

Highlights:

• The odoc team is starting work on improving odoc performances. After shipping important features, including a link to source code, syntax for tables, and recent support for search, they are turning their focus on consolidating the full documentation generation stack (including Dune rules and integration with OCaml.org package docs) and improving performances.
• opam 2.2~alpha3 is out! It is the last alpha release, and the opam team plans to start the beta release next cycle.
• Anticipating on the January update, the compiler PR necessary for Merlin project-wide references has been merged! This will be part of OCaml 5.2, meaning that starting with the next compiler version, OCaml developers will be able to query for project-wide references (and limited support for project-wide rename!) in their projects. Read more here to know how you can test the feature.

Releases:

• Odoc 2.4.0
• OCaml LSP 1.17.0
• Merlin 4.13
• Dune 3.12.1
• opam 2.2.0~alpha3
  

[Dune] Exploring Package Management in Dune (W4)

Contributed by: @rgrinberg (Tarides), @Leonidas-from-XIV (Tarides), @gridbugs (Tarides), @kit-ty-kate (Tarides), @Alizter

Why: Unify OCaml tooling under a single command line for all development workflows. This addresses one of the most important pain points reported by the community.

What: Prototyping the integration of package management into Dune using opam as a library. We're introducing a dune pkg lock command to generate a lock file and enhancing dune build to handle dependencies in the lock file. More details in the Dune RFC.

Activities:

• Introduced a locking mechanism to prevent conflicts in multiple repositories and fixed a bug in Dune's locking code. – ocaml/dune#9140
• Enabled project locking without a network connection using locally cached opam-repository. – ocaml/dune#9202
• Enhanced handling of opam-repositories with non-standard contents (non-file objects). – ocaml/dune#9352
• Added a feature where users can set arbitrary variables which can be referred to by opam packages while solving a project's dependencies. This gives users more control over decisions made by the solver and which dependencies their project ends up having in its lockdir. – ocaml/dune#9325
• Ensured lockdir contains all dependencies of local packages for consistency. This prevents Dune from presenting inconsistent information to users who have changed their project's dependencies after creating a lockdir. Instead, they'll be prompted to recompute their lockdir. – ocaml/dune#9156
• Ability to detect which dependencies in the lockdir are only needed when building tests. This will be necessary to allow users to skip downloading and building packages that are only needed for testing unless they are running tests. – ocaml/dune#9095
• Improved support for Windows, focusing on better Curl integration and opam library adjustments for Windows architectures. – ocaml/dune#9252, ocaml/dune#9048
• Refined context/lock file handling in Dune package management. – ocaml/dune#9343
• Refined context/lock file handling. Before each context had an associated lock file so lock files were selected via context arguments. Now the context middlemen have been removed and the users selects lock files by specifying the lockfile itself. This applies to most pkg related commands. – ocaml/dune#9343
• Added support for the conflicts field in opam files – ocaml/dune#9340
• Looked into solutions to add support for deptops – ocaml/dune#9430
• Writing the PID of process that created the lock file ocaml/dune#9295 to be able to easily determine which process is holding the revision store lock
• Avoid Git translating its CLI ocaml/dune#9390. Since we use the Git binary under the hood, users with different locales might get translated Git output. Since our Git output is not shown to the user, we disable translation.
• Remove opam-repository-url option ocaml/dune#9373 Removes all CLI options that deal with repositories. All opam-repositories are now controlled in the dune-workspace file.
• Support for specifying particular branches/commits for repos ocaml/dune#9241 adds support for specifying branches and commits as opam-repository sources, so the user can fix one particular state of opam-repository
• Enable checking out tags ocaml/dune#9471 adds the same support as above but for tags. It does so by saving the tags in per-remote namespaces, thus it works very much like branches.
• Implement downloading sources via Git ocaml/dune#9506 enables cloning source directories via the rev store, thus caching most commits when working with multiple projects from the same repo, etc.
• Read the main branch correctly even if tags exist ocaml/dune#9549 fixes an issue where the code that determines the tracking branch got confused in the presence of namespaced tags.
• Dune will compute checksums for lock files of packages which don't already have checksums in their opam metadata. – ocaml/dune#9384
• Package metadata for a Dune project can be read from .opam files rather than dune-project – ocaml/dune#9418
• Support for conflict classes in opam files for the solver – ocaml/dune#9442
• The ability to add additional constraints to feed the solver – ocaml/dune#9337

[opam] Native Support for Windows in opam 2.2 (W5)

Contributed by: @rjbou (OCamlPro), @kit-ty-kate (Tarides), @dra27 (Tarides), @AltGr (OCamlPro)

Why: Enhance OCaml's viability on Windows by integrating native opam and opam-repository support, fostering a larger community and more Windows-friendly packages.

What: Releasing opam 2.2 with native Windows support, making the official opam-repository usable on Windows platforms.

Activities:

• We’ve released opam 2.2.0~alpha3, a culmination of the last 4 months of work. This release, amongst other fixes and improvements, adds a new x-env-path-rewrite field necessary for Windows support. - #5636
• We’ve also fixed a number of issues on Windows:
  • Fix issues in the C stubs for Windows - #5714
  • Fix incorrect error message in configure on Windows - #5667
  • Always resolve the fullpath to the cygpath executable - #5716

[odoc] Add Search Capabilities to odoc (W25)

Contributed by: @panglesd (Tarides), @EmileTrotignon (Tarides), @julow (Tarides), @jonludlam (Tarides)

Why: Improve usability and navigability in OCaml packages documentation, both locally and on OCaml.org, by offering advanced search options like type-based queries.

What: Implementing a search engine interface in odoc, complete with a UI and a search index. Additionally, we're developing a default client-side search engine based on Sherlodoc.

Activities:

• After merging the PR that added support for search to odoc in October, we continued work on building an odoc-compatible search engine based on Sherlodoc. This will give a search engine with type-based search for every package that uses odoc. The plan is to make Sherlodoc the search engine in Dune's documentation generation. You can try an early demo on Varray's doc – art-w/sherlodoc#4
• Buiding on odoc's support for search, we merged a PR that adds occurrences information to the search index. This will allows odoc search engines to improve the order of search results by using the number of occurences. – ocaml/odoc#976

[odoc] Syntax for Images and Assets in odoc (W25)

Contributed by: @panglesd (Tarides), @jonludlam (Tarides), @dbuenzli, @gpetiot (Tarides)

Why: Empower package authors to create rich, engaging documentation by enabling the integration of multimedia elements directly into OCaml package documentation.

What: We're introducing new syntax and support for embedding media (images, audio, videos) and handling assets within the odoc environment.

Activities:

• Added assets in the environment to treat them similarly as other resolvable elements. This addressed the remaining feedback from reviews, and if no other blocker is found, the PR should be ready to merge. – ocaml/odoc#1002

[odoc] Improving odoc Performance (W25)

Contributed by: @jonludlam (Tarides), @julow (Tarides), @gpetiot (Tarides)

Why: Address performance issues in odoc, particularly for large-scale documentation, to enhance efficiency and user experience, and unlock local documentation generation in large code bases.

What: Profiling odoc to identify the main performance bottlenecks, and optimising odoc with the findings.

Activities:

• Experimented with different data structures and algorithms for more efficient documentation generation on large files. –ocaml/odoc#1033, ocaml/odoc#1036, ocaml/odoc#1049
• Implemented item lookup improvements in signatures for faster processing. – ocaml/odoc#1049
• Developed a fix for memory issues caused by module type of expressions, with promising results from testing at Jane Street. – ocaml/odoc#1042

[Dune] Generate Dependencies Documentation with Dune (W25)

Contributed by: @jonludlam (Tarides)

Why: Enhance the usability of locally-generated documentation by providing direct access to dependencies' documentation.

What: Implementing new Dune rules for odoc to enable efficient documentation generation and access to documentation for all opam packages in your switch.

Activities:

• The new Dune rules have been merged and are available in the newest release of Dune 3.12.1. Try running dune build @doc-new to generate your documentation and tell us what you think! – ocaml/dune#8803
• Shortly after the release, we noticed an issues with dependency handling. We're working on a fix that should be released – ocaml/dune#9461

[Merlin] Support for Project-Wide References in Merlin (W19)

Contributed by: @voodoos (Tarides), @trefis (Tarides), @Ekdohibs (OCamlPro), @gasche (INRIA)

Why: Enhance code navigation and refactoring for developers by providing project-wide reference editor features, aligning OCaml with the editor experience found in other languages.

What: Introducing merlin single occurrences and LSP textDocument/references support, extending compiler's Shapes for global occurrences, and integrating these features in Dune, Merlin, and OCaml LSP.

Activities:

• The first iteration on project-wide occurrences is closing-in, and we made a custom opam-repository to test the feature while the changes make their way into the upstream compiler. This gave us the opportunity to test the feature on more real projects, and after another round of bug-fixing and UI improvement we opened it to the community for wider testing.
• Anticipating the January update, the compiler PR has been merged! The next steps are to revisit the patches on the other projects and to open PRs upstream. Dune is the next in line. – ocaml/ocaml#12508

[Merlin] Improving Merlin's Performance (W19)

Contributed by: @pitag (Tarides), @Engil (Tarides), @3Rafal (Tarides)

Why: Some Merlin queries have been shown to scale poorly in large codebases, making the editor experience subpar, and users report that they sometimes must wait a few seconds to get the answer. This is obviously a major issue that hurts developer experience, so we're working on improving Merlin performance when it falls short.

What: Developing benchmarking tools and optimising Merlin's performance through targeted improvements based on profiling and analysis of benchmark results.

Activities:

• We've made the file cache lifetime configurable. Varying the lifetime will allow experimenting with Merlin's time/space trade-off. Before, it was constantly set to 5 min. -#1698
  • Introduced a config that lets users set a file cache lifespan. When modified to a larger value it should improve performance for large repositories – ocaml/merlin#1698
  • Introduced the file cache lifespan flag to ocaml-lsp, so it can be used for all LSP clients – ocaml/ocaml-lsp#1210
• We've enriched the telemetry that comes embedded in the ocamlmerlin responses:
  • We've added information about cache hits and misses of the various Merlin caches: cmi-files cache, cmt-files cache, typer phase cache, PPX phase cache, and reader phase cache. - #1711
  • We've added information about the size of the major heap at the end of an ocamlmerlin query - #1717
• We've finished up Merlin's new Fuzzy CI, a by-product of the performance work.
  • We've opened the PRs - #1716 (and #1719)
  • We've written a GitHub wiki entry with a high-level description about it. - Merlin Fuzzy CI