## Goal

The Sandmark benchmarking runs are currently executed nightly using a JSON configuration file. The goal is to use a bot to trigger benchmark runs on PRs on a need basis.

## KRs

We require a new KR for this work item.

## How to measure this KR

The number of OCaml/OCaml PRs that will use the bot to trigger benchmamrk runs is a good metric.

## Design

### coqbot

The Github [pull request webhook](https://docs.github.com/en/webhooks-and-events/webhooks/webhook-events-and-payloads#pull_request) will allow use of coq bot to listen to commands. For a specific `run-benchmark` command in a PR, the bot should trigger the Sandmark benchmark run for the PR. We will need to scope out the workflow in using the backend OCurrent pipeline to trigger the benchmark execution. After the Sandmark nightly run finishes, the results of the Sequential and Parallel `Normalized Time` compared with trunk will be posted on the respective PR with permalinks to the dashboard.

## Scope

The inclusion of a new command in coqbot is essential to support triggering of the benchmark runs. The OCluster workers can currently only build Docker or OBuilder jobs. So, we will have to explore the feasibility to run the Sandmark benchmarks natively. Also, we will need to move away from the Sandmark nightly setup and configuration.

## Challenges

Since we already have a production deployed environment, we need to take an incremental approach to use the bot:
1. Enable reporting back results from Sandmark to PRs that were added to the configuration.
2. Enable automatically picking up which PRs are to be benchmarked (still nightly), using a tag or a GitHub comment.
3. Try to move away from nightly runs and try to use the benchmarking machines more optimally.

## Alternative designs considered

### Updating sandmark-nightly-config

We initially considered updating the sandmark-nightly-config file for a new benchmark trigger. In order to standardize the efforts of using OCurrent as the defacto implementation for all benchmarking for OCaml projects, we have decided against this implementation.

## Deliverables

1. Use of a new coq bot command to trigger Sandmark benchmark runs.
2. Support in OCurrent pipeline to trigger benchmark exeuctions.
3. Posting of benchmark results in a tabular format on GitHub PRs.

## Effort

* 1 Senior Software Engineer * 3 weeks

### Justification

The use of Coq bot to trigger Sandmark runs need to be tested using the GitHub App. We will also need to explore the use of OCurrent pipelines for the same, and hence the integration points need to be checked. Also, the posting of the results of the Sandmark benchmark runs in a tabular format is to be implemented.

Source: https://github.com/coq/bot/issues/288

## Priority

High

### Why?

The performance of the OCaml compiler needs to be tracked periodically for any regressions. It is a key characteristic for the compiler, and the Benchmark Tooling team must facilitate the benchmark runs as needed by compiler developers.

## Timeline

Work starts in Q3 2023 and is expected to finish by Q4 2023.

## Point of Contact

Puneeth Chaganti, Ambre Suhamy, Shakthi Kannan, Riku Silvola.

## Slack Channel

#sandmark-dev on https://ocamllabs.slack.com

## Approvals

* Sadiq Jaffer
* Anil Madhavapeddy
* Florian Angeletti