# current-bench: 2023 and Beyond

## Abstract

The current-bench project provides the infrastructure to run predictable, continuous benchmarking for OCaml projects. It is being used for benchmarking Dune, Irmin, and Merlin projects. We have re-written the current ReScript frontend with the new Dream web framework, and have also started to add benchmarks for many OCaml projects such as Eio, odoc, kcas, lockfree, etc. In this talk, we would like to present the new current-bench features, OCurrent pipeline with OCluster, system configuration changes, and also share our experience on onboarding new OCaml projects for benchmarking.

## current-bench Features

The current-bench pipeline has added support for new custom OCluster workers to build and run benchmarks from the main pipeline. You can also define a default worker to be used in the configuration. With the release of OCaml 5 and multicore support, you can run benchmarks on multiple CPU cores by specifying the Docker CPUs. An API has also been implemented to allow directly posting the JSON results to current-bench. In addition to supporting the use of a Makefile in an OCaml project with a `make bench` target, you can include a bench.Dockerfile that provides the necessary steps to run the benchmarks. The UI usability has been improved by providing a direct link to GitHub for the PRs, showing minimum, maximum and average values, allowing a description for the metrics, indicating if the metric is ascending or descending, supporting overlay graphs, showing multi-value data points, displaying execution time for functions etc. Prometheus monitoring support and Slack alert notifications have been integrated with current-bench. The JSON parsing code has been completely refactored with improved error handling. The base Docker image has also been updated to use OCaml 5.0.0. The latest current-bench dashboard is shown in the figure.


## Experience Report

The current-bench UI uses ReScript - a simple, fully typed language for the frontend. We have roughly twice more ReScript frontend code (5.7k loc) than OCaml backend code (3k loc), even though ReScript is doing only plumbing without using any advanced features. Also, ReScript encourages a less safe approach with its standard library and interoptability with JavaScript. It is very hard to track the bugs that result from OCaml with JavaScript semantics. Moreover, we have to rewrite the SQL backend queries in GraphQL, which in turn generates a different data model and makes it harder to share the common logic with the backend. This is quite challenging when we want to extend our database schema model. We did try to share code between the backend and the frontend, but it has its own limitations - ReScript is stuck on OCaml 4.06, and the shared code can only depend on the standard 4.06 library. Also, ReScript brings with it a large number of dependencies with NPM. We have been able to rewrite the frontend using the Dream web framework, which is easy to use, boilerplate-free web framework that supports TLS, WebSockets and GraphQL. The code base is now stable, easy to maintain and can be extended for future use.

We have also been working with OCaml project maintainers to add benchmarks to their respective projects, so that the code performance can be continuously measured with current-bench. We continue to add features and enhancements to the pipeline to provide a better developer experience. For example, to run the OCaml/OCaml compiler benchmarks on a weekly basis, we have added support to use a separate benchmarks repository that can be specified using a configuration variable. A new cb-checker tool has been implemented to validate the parsing of JSON output from the benchmark results without having to integrate with current-bench. Using this tool, any maintainer of an OCaml project can ensure that their output benchmark result in JSON format can be parsed by current-bench. A staging server is setup to test run the benchmarks for new benchmarks, before maintainers can proceed to install the GitHub App required to run current-bench for the project. This is to avoid any benchmarks from directly running and crashing the main pipeline production deployment. At present, the current-bench graphs are time-series, but we have requests for custom visualisations such as distribution plots, box plots, heat maps etc. During the onboarding phase, we do work closely with the developers to get their benchmarks tested and their projects added to current-bench.

## Future Work

In future, we would like to integrate the benchmarking pipeline with the Continuous Integration (CI) workflow so that the developers can immediately get the results of the benchmarks as soon their CI build and tests succeed. The current-bench infrastructure tracks performance regressions on Intel and AMD x86 machines only, and there are requests to run on ARM hardware for comparisons. The `bench` target in the Makefile can be replaced by moving to dune, and this can be executed with `dune bench`. The support for using opam lockfiles to fix the dependency version needs to be addressed. We definitely will need to add support for additional visualisations and graphs in current-bench apart from time-series analysis. There is considerable interest to view SPECpower and energy measurements using tools like perf, ipmitool, variorum etc. We will also need to work on adding more domain-specific benchmarks that are of interest to various communities.

## References

[Gar21] Gargi Sharma, Rizo Isrof, Magnus Skjegstad. Continuous Benchmarking for OCaml Projects.

[Kel20] Tom Kelly. Hardware and OS setting for Linux benchmarking. https://github.com/ocaml-bench/ocaml_bench_scripts#notes-on-hardware-and-os-settings-for-linux-benchmarking. 2019 (accessed May 2023)