# Multiple collectors for rustc-perf

> The overall design of the benchmark suite is described [here](https://kobzol.github.io/rust/rustc/2023/08/18/rustc-benchmark-suite.html), reading that post might be useful before reading this document if you don't know how `rustc-perf` works.

Currently, the [Rust benchmark suite](https://github.com/rust-lang/rustc-perf) (rustc-perf) uses a single machine (*collector machine*) to execute benchmarks. This is limiting because of two reasons:

1. Running the benchmarks takes a relatively long time. The longer this is, the slower is the feedback that we provide to compiler contributors. While the current benchmark run duration (~1.25h) is not *that bad*, it can add up when multiple PRs get into the queue. But perhaps more importantly, we would actually like to get more data per benchmark run (benchmark different codegen backends, frontend thread counts, perform more benchmark iterations, etc.), but we did not do that yet because of the single collector machine bottleneck.
2. We currently only gather benchmark data on the x64 Linux platform. While this is the most popular Rust platform at the moment, it would be interesting to see the performance effect of compiler changes also for other hardware architectures and/or operating systems (e.g. ARM, in addition to the current x64 architecture). This is obviously not possible with just a single collector machine.

This document thus tries to write down what would it take to add support for multiple collector machines to rustc-perf, so that multiple benchmarks can run in parallel. It enumerates the design constraints that we should adhere to and also tries to suggest some specific solutions that could be used to achieve the goal.

## How the components communicate together
In simplified terms, collection of benchmarks currently works like this:
- The collector machine runs an [endless loop](https://github.com/rust-lang/rustc-perf/blob/master/collector/collect.sh), in which it pulls the latest version of `rustc-perf` from git, and then runs the `collector bench_next` command.
- The `bench_next` command connects to [perf.rlo](https://perf.rust-lang.org/) and asks it what artifact (compiler commit SHA) it should benchmark next using the `/perf/next_artifact` endpoint. The website maintains an internal queue, and returns the compiler SHA of the next entry in the queue that should be benchmarked. If there is nothing to benchmark, the collector machine sleeps for a few minutes and then tries again.
- If a compiler SHA (or version tag) is returned, the collector machine downloads a corresponding compiler toolchain (using `rustup` or directly from S3 storage).
- The collector figures out what benchmarks to run by loading a set of benchmark directories from disk. Then it marks the start of the benchmark collection in the DB by storing benchmark steps (the individual benchmarks that will be executed) into the `collector_progress` table. This is then used by perf.rlo's [status page](https://perf.rust-lang.org/status.html) to report the progress of the collection.
- When a benchmark step is finished, the `collector_progress` table is updated.
- When all benchmark steps are finished, the collector machine writes an entry into the `artifact_collection_duration` table, which marks the whole benchmark collection as finished, and records its duration.
- After finishing the collection, the collector machine sends a request to `/perf/onpush`. This forces the website to break its cache and update the list of benchmarked artifacts from the DB. The website also automatically marks any PRs that were queued, and are not currently in-progress, as finished, and posts a comment to the corresponding PR.

This approach has been working relatively fine so far, and it is rather simple; the collector just runs an endless loop and always gets a single artifact that it benchmarks from start to finish. However, it is not tested in any way, it is full of undocumented invariants and does not handle edge cases well. For example:
- If the collection of a certain benchmark step starts, and then crashes in the middle, it will be skipped the next time, and potentially missing configurations won't be benchmarked.
- When a master PR is returned from `/perf/next_artifact`, it is stored as being queued in the DB. When the collection crashes, and then a new master PR gets ahead of the previous unfinished PR, after the next collection finishes, both of them will be (wrongly) marked as completed.
- If a compiler SHA gets into the queue/DB, but its toolchain cannot be downloaded (e.g. because a CI workflow that was supposed to upload the toolchain somehow failed or we force-pushed to rustc's master), the DB needs to be manually pruned, because the collector machine will get stuck in an endless loop trying to download the toolchain.

Luckily, collection usually doesn't crash, so we don't have to deal with these problems often.

## How the queue works
TODO

## Design constraints
Here is a set of design constraints that we would like the resulting (multi collector machine setup) to have. Some of them are shared with the current implementation, while some of them are added as new ones. Some of them are more "must-have", some of them "nice-to-have".

- **Simplicity** The system should be easy to understand, and primarily easy to maintain. We cannot devote a lot of resources to babysit rustc-perf, and only very few people have access to its deployment on AWS. Ideally, it should be possible to test the multi-collector benchmark setup locally, without using third-party solutions that require authentication, so that people can debug what's going on and simulate various situations even if we don't necessarily have automated end-to-end tests.
- **Robustness against errors**. Currently, we can stop the collector machine or the perf.RLO website pretty much whenever, and then once they are restarted, the whole system is able to make progress. The latest collection will be restarted from scratch, but the system does not get stuck. The system should be resilient towards e.g. one of the collector machines being temporarily turned off, and perhaps even allow reporting partial results to users in that case, to avoid the system being stuck even though some results could already be reported.
- **Keep critical path as short as possible**. The critical path is the total duration of the whole benchmark collection, after which the results are reported to the user. There are certain collector machine configurations that can reduce it, for example if we add a new collector machine for the same architecture (x64), in which case the critical path can be reduced by a half (approximately). On the other hand, if we e.g. add a new ARM machine, the critical path will remain the longer of the durations to benchmark x64 and ARM. We might want to keep the number (and approximate performance) of the machines for different architectures similar, otherwise there might not be many benefits for users when adding a new machine to the mix. In other words, if we always wait for the full result before reporting it, we assume benchmarking the same set of configurations for all hardware architectures, and we have e.g. two x64 machines, but only one ARM machine, then the second x64 machine is likely to be useless.
- **Updatability of rustc-perf code on the collector**. Currently, the collector machine tries to update its own code by doing `git pull` and building `rustc-perf` after every performed benchmark collection. We should make sure that the collector machines can still update themselves, and ideally that they all run the same code. It would not be great if a benchmark collection is performed with machine A being on commit C1 and machine B being on commit C2.
- **Stable benchmark results**. The distributed nature of benchmarking is complicated by the fact that we would ideally like to always execute the same benchmarks on the same collector machine, if at all possible. There are two levels of this support:
- Always execute a given benchmark on the same collector machine, even when *new* machines are added in the future. This is probably not needed and is unnecessarily strict.
- Given a steady set of collector machines, always execute a given benchmark on the same collector machine. This should help keep the benchmark results as stable as possible, because even if the machines seem identical, there will always be some differences between them.

This constraint will require us to figure out how to split the benchmark suite so that a given collector machine always runs the same benchmarks. We will probably need to identify the collector machines somehow and then assign these identities to individual benchmarks. This can get all kinds of tricky; it will be discussed in more detail in the [Possible approaches](#Possible-approaches) section.
- **Management of collector machines**. Once we start having more than a single machine, their management (observability, restarts, OS updates, configuration to get a low-noise benchmarking environment) will become more difficult. It would be nice to use some solution to make this easier, e.g. using Ansible or some other similar technology, to avoid us having to SSH into N machines in order to (re)configure something.
- **Observability**. Currently, we have logs from perf.rlo on AWS, and we can access logs from the collector machine through systemd (we just SSH to the machine and inspect the logs locally). Once multiple machines are added, observing what's going on will become tricky. It would be nice to setup some solution for that, e.g. Grafana/Prometheus or something that will allow us to see the logs from all collector machines at a single place. A related aspect is that it should be possible to observe the progress of the benchmark through the status page, same as it is now. Ideally, it should be possible to observe the status of the individual collector machines on the status page.
- **Local execution**. rustc-perf is being used by compiler contributors locally, and in this case there should not be any complicated distributed setup that would need to be configured. It should be possible to run the collector's `bench_local` command on a local SQLite DB and then visualize the results in CLI or the website with as little hassle as possible.
- **Testability**. It would be great if we had some tests for the distributed benchmarking architecture itself, to test some known edge cases. Relatedly, a staging environment would go a long way towards testing the system and making sure that changes will not break the production deployment.

## Possible approaches
See https://hackmd.io/@davidtwco/ByZ0ox5K1l
<!--
This section describes some possible approaches to add support for multiple collector machines to rustc-perf, given the design constraints above. The point is not to get deep into specifics, but just spitball a few high-level approaches, mostly differing in the technologies that they use and the complexity that they would bring.

### Use the existing Postgres DB as a benchmark queue
Instead of the collector deciding which benchmarks to run, the server would have that responsibility. When it gets a new benchmark request, it would proactively push a set of benchmarking jobs into the DB. The collector would then pop these jobs from the DB, benchmark them, and then mark them as completed. Once all jobs from a collection are completed, the server would mark the benchmark as finished.

- Each collector machine will get an associated ID (numeric or string, doesn't really matter).

- TODO: How to handle priority when things are added to the DB proactively?

- Each collector machine should have its associated (string?) ID. Modify the collector CLI so that it is possible to specify this ID, especially for the `bench_next` command (for the local commands, it can be just set automatically to some predetermined value).
- Modify the DB so that we are able to store which collector machine performed which step of the artifact benchmark collection. Ideally by adding a link to the [`collector_progress`](https://github.com/rust-lang/rustc-perf/blob/master/database/schema.md#collector_progress) table which will reference a table row that describes the collector machine (a new DB table should be created for this). A more simple approach would be to just identify the machine with a string and just store that string directly in `collector_progress`.
- Implement support for multiple collector machines benchmarking the same artifact. Probably the simplest way of achieving that is to use this logic:
- One *primary* process will enqueue all the benchmark steps that need to be performed into the DB (the `collector_progress` table). Then, each collector machine will scan this table and execute any benchmarks (steps) that are not yet finished and that are assigned to its collector machine ID.
- This *primary* process can be the `site` binary itself. This would simplify the collectors, at the cost of duplicating/moving some benchmark resolution logic to the site, and also requiring the site to have access to the benchmarks (currently the site is a self-contained binary, and it reads benchmark metadata during compilation, not while it is executing).
- Or, the *primary* can be one of the collector machines, which will prefill the `collector_progress` table, and then wait until all the benchmark steps are completed. It can either be a separate command that does this synchronization, or it can just be the `bench_next` command, in which case the *primary* would be executing benchmarks by itself.
- After this change is performed, modify the notification logic in the website (`/perf/onpush`) so that it can correctly recognize that the benchmark has been finished.
-->