Fast and Curious: Benchmarking (Multicore) OCaml

# Fast and Curious: Benchmarking (Multicore) OCaml ## Summary An oft overlooked aspect of programming language development process is benchmarking. While popular CI/CD tools such as Azure pipelines, Gitlab CI, Circle CI, Drone.io and Github Actions are regularly used for continuous testing and deployment, there is a dearth of such tools for continuous benchmarking. This is because benchmarking well is challenging due to the care needed to choose appropriate benchmarks, the complexity of modern OSes and hardware that makes it hard to obtain reproducible results, and a powerful dashboard needed to surface resultant metrics in a useful manner that can be explored interactively by the developer. For benchmarking the compiler of the OCaml [1] programming language, we have developed Sandmark [2]. Sandmark is a (a) suite of well chosen benchmarks aimed at exercising different performance axes including CPU, memory and IO (b) a tool to run the benchmarks, building the compiler and its dependencies under varying configuration settings and (c) a dashboard [3] to interactively explore the results of the benchmark runs. Sandmark was originally developed for supporting the Multicore OCaml [4] project, which aims to add native support for concurrency and parallelism to OCaml, and has undergone significant improvements since the initial release [5]. We have learnt several useful lessons in building a continuous benchmarking infrastructure that we would like to share in this talk, which may benefit developers who work on projects where performance is critical. In this talk, I will describe the Sandmark tool, illustrate the quirks of modern OSes and hardware and how we overcame them, highlighting useful takeaways for setting up continuous benchmarking for your own projects. ### References [1] OCaml https://ocaml.org/ [2] Sandmark https://github.com/ocaml-bench/sandmark [3] sandmark-nightly: https://github.com/ocaml-bench/sandmark-nightly [4] Multicore OCaml: https://github.com/ocaml-multicore/ocaml-multicore/ [5] Tom Kelly. Benchmarking the OCaml compiler: our experience, OCaml Workshop 2019, https://icfp19.sigplan.org/details/ocaml-2019-papers/3/Benchmarking-the-OCaml-compiler-our-experience ## Outline/Structure of the Demonstration * Continuous Benchmarking (5 minutes) - The Challenge - Present-day Benchmarking * What is Sandmark (5 minutes) - Introduction - Setup - Configuration - Features - Tuning * Experience Report (10 minutes) - Case Studies - Conclusion and future work ## Learning Outcome * Continuous benchmarking for programming language implementations. * Learn to use Sandmark for running benchmarks for OCaml and Multicore OCaml. * Tuning of hardware for Multicore parallel programs. ## Target Audience * Programming language enthusiasts who are interested in performance analysis. ## Prerequisites for Attendees None.