The overall design of the benchmark suite is described here, reading that post might be useful before reading this document if you don't know how
rustc-perfworks.
Currently, the Rust benchmark suite (rustc-perf) uses a single machine (collector machine) to execute benchmarks. This is limiting because of two reasons:
This document thus tries to write down what would it take to add support for multiple collector machines to rustc-perf, so that multiple benchmarks can run in parallel. It enumerates the design constraints that we should adhere to and also tries to suggest some specific solutions that could be used to achieve the goal.
In simplified terms, collection of benchmarks currently works like this:
rustc-perf from git, and then runs the collector bench_next command.bench_next command connects to perf.rlo and asks it what artifact (compiler commit SHA) it should benchmark next using the /perf/next_artifact endpoint. The website maintains an internal queue, and returns the compiler SHA of the next entry in the queue that should be benchmarked. If there is nothing to benchmark, the collector machine sleeps for a few minutes and then tries again.rustup or directly from S3 storage).collector_progress table. This is then used by perf.rlo's status page to report the progress of the collection.
collector_progress table is updated.artifact_collection_duration table, which marks the whole benchmark collection as finished, and records its duration./perf/onpush. This forces the website to break its cache and update the list of benchmarked artifacts from the DB. The website also automatically marks any PRs that were queued, and are not currently in-progress, as finished, and posts a comment to the corresponding PR.This approach has been working relatively fine so far, and it is rather simple; the collector just runs an endless loop and always gets a single artifact that it benchmarks from start to finish. However, it is not tested in any way, it is full of undocumented invariants and does not handle edge cases well. For example:
/perf/next_artifact, it is stored as being queued in the DB. When the collection crashes, and then a new master PR gets ahead of the previous unfinished PR, after the next collection finishes, both of them will be (wrongly) marked as completed.Luckily, collection usually doesn't crash, so we don't have to deal with these problems often.
TODO
Here is a set of design constraints that we would like the resulting (multi collector machine setup) to have. Some of them are shared with the current implementation, while some of them are added as new ones. Some of them are more "must-have", some of them "nice-to-have".
Simplicity The system should be easy to understand, and primarily easy to maintain. We cannot devote a lot of resources to babysit rustc-perf, and only very few people have access to its deployment on AWS. Ideally, it should be possible to test the multi-collector benchmark setup locally, without using third-party solutions that require authentication, so that people can debug what's going on and simulate various situations even if we don't necessarily have automated end-to-end tests.
Robustness against errors. Currently, we can stop the collector machine or the perf.RLO website pretty much whenever, and then once they are restarted, the whole system is able to make progress. The latest collection will be restarted from scratch, but the system does not get stuck. The system should be resilient towards e.g. one of the collector machines being temporarily turned off, and perhaps even allow reporting partial results to users in that case, to avoid the system being stuck even though some results could already be reported.
Keep critical path as short as possible. The critical path is the total duration of the whole benchmark collection, after which the results are reported to the user. There are certain collector machine configurations that can reduce it, for example if we add a new collector machine for the same architecture (x64), in which case the critical path can be reduced by a half (approximately). On the other hand, if we e.g. add a new ARM machine, the critical path will remain the longer of the durations to benchmark x64 and ARM. We might want to keep the number (and approximate performance) of the machines for different architectures similar, otherwise there might not be many benefits for users when adding a new machine to the mix. In other words, if we always wait for the full result before reporting it, we assume benchmarking the same set of configurations for all hardware architectures, and we have e.g. two x64 machines, but only one ARM machine, then the second x64 machine is likely to be useless.
Updatability of rustc-perf code on the collector. Currently, the collector machine tries to update its own code by doing git pull and building rustc-perf after every performed benchmark collection. We should make sure that the collector machines can still update themselves, and ideally that they all run the same code. It would not be great if a benchmark collection is performed with machine A being on commit C1 and machine B being on commit C2.
Stable benchmark results. The distributed nature of benchmarking is complicated by the fact that we would ideally like to always execute the same benchmarks on the same collector machine, if at all possible. There are two levels of this support:
This constraint will require us to figure out how to split the benchmark suite so that a given collector machine always runs the same benchmarks. We will probably need to identify the collector machines somehow and then assign these identities to individual benchmarks. This can get all kinds of tricky; it will be discussed in more detail in the Possible approaches section.
Management of collector machines. Once we start having more than a single machine, their management (observability, restarts, OS updates, configuration to get a low-noise benchmarking environment) will become more difficult. It would be nice to use some solution to make this easier, e.g. using Ansible or some other similar technology, to avoid us having to SSH into N machines in order to (re)configure something.
Observability. Currently, we have logs from perf.rlo on AWS, and we can access logs from the collector machine through systemd (we just SSH to the machine and inspect the logs locally). Once multiple machines are added, observing what's going on will become tricky. It would be nice to setup some solution for that, e.g. Grafana/Prometheus or something that will allow us to see the logs from all collector machines at a single place. A related aspect is that it should be possible to observe the progress of the benchmark through the status page, same as it is now. Ideally, it should be possible to observe the status of the individual collector machines on the status page.
Local execution. rustc-perf is being used by compiler contributors locally, and in this case there should not be any complicated distributed setup that would need to be configured. It should be possible to run the collector's bench_local command on a local SQLite DB and then visualize the results in CLI or the website with as little hassle as possible.
Testability. It would be great if we had some tests for the distributed benchmarking architecture itself, to test some known edge cases. Relatedly, a staging environment would go a long way towards testing the system and making sure that changes will not break the production deployment.
-
Any changes
Be notified of any changes
-
Mention me
Be notified of mention me
-
Unsubscribe
Subscribe