# rustc-perf benchmarks upgrade 2022

As per [the roadmap](https://hackmd.io/YJQSj_nLSZWl2sbI84R1qA?view), we want to improve the rustc-perf benchmark suite.

Task completion is indicated like so:
- [ ] incomplete
- [x] complete

**As of 2022-04-29, the upgrade is complete!**

Notes:
- Real-world crates include the version number in their name, for clarity, and to allow for future updates.
- No pre-release versions, just proper release, to avoid annoying version numbers like `1.2.3-pre.1`.
- For dates below: there was a big benchmarks rearrangement on 2017-11-{02,03}, and getting version history before then is a pain, so I haven't gone back farther than that. That's plenty old.
- New benchmarks should be given `.patch` files to test `incr-patched`.

## Primary

Ones marked with `*` are part of the "stable" set (see below).

Easy changes. (Latest version numbers were correct at time of writing, but may have changed.)

See the instructions at the bottom of this document for the update procedure.

- [x] `cargo` (0.29, 2018-05-23) -> `cargo-0.60.0` \[nnethercote, [#1209](https://github.com/rust-lang/rustc-perf/pull/1209), [#1211](https://github.com/rust-lang/rustc-perf/pull/1211)\]
- [x] `clap-rs` (2.29.0, 2018-02-25) -> `clap-3.1.6` \[nnethercote, [#1217](https://github.com/rust-lang/rustc-perf/pull/1217), [#1235](https://github.com/rust-lang/rustc-perf/pull/1235)\]
- [x] `cranelift-codegen` (0.30.0, 2019-04-11) -> `cranelift-codegen-0.82.1` [lqd, [#1228](https://github.com/rust-lang/rustc-perf/pull/1228), [#1247](https://github.com/rust-lang/rustc-perf/pull/1247)]
- [x] `diesel` (2.0.0??, 2020-12-03) -> `diesel-1.4.8`
- Version number in `collector/benchmarks/diesel/diesel/Cargo.toml` says 2.0.0, weird [lqd, [#1238](https://github.com/rust-lang/rustc-perf/pull/1238), [#1248](https://github.com/rust-lang/rustc-perf/pull/1248)]
- [x] `futures`* (0.1.0, pre-2017-11-03) -> `futures-0.3.21`
- The `futures-0.3.21` crate is now just a facade crate with a bunch of sub-crates: `futures-core`, `futures-i/o`, etc. Because we only measure leaf crate compilation, it's boring and not that different to `helloworld`. So we can just move the existing `futures` crate (which does have actual code in it) to the stable set. \[nnethercote, [#1218](https://github.com/rust-lang/rustc-perf/pull/1218)\]
- [x] `helloworld` (unchanged)
- [x] `hyper-2` (0.13.0-alpha.4, 2019-11-20) -> `hyper-0.14.18` [lqd, [#1251](https://github.com/rust-lang/rustc-perf/pull/1251), [#1262](https://github.com/rust-lang/rustc-perf/pull/1262)]
- [x] `piston-image`* (0.10.3, pre-2017-11-02) -> `image-0.24.1` [rylev, [#1229](https://github.com/rust-lang/rustc-perf/pull/1229)]
- Plus keep the old one as part of the stable set. ([#1234](https://github.com/rust-lang/rustc-perf/pull/1234))
- [x] `regex`* (0.1.80, pre-2017-11-02) -> `regex-1.5.4` [rylev, [#1231](https://github.com/rust-lang/rustc-perf/pull/1231), [#1239](https://github.com/rust-lang/rustc-perf/pull/1239)]
- Plus keep the old one as part of the stable set.
- [x] `ripgrep` (0.8.1, 2018-05-23) -> `ripgrep-13.0.0` \[Kobzol, [#1224](https://github.com/rust-lang/rustc-perf/pull/1224), [#1232](https://github.com/rust-lang/rustc-perf/pull/1232)\]
- [x] `serde` (1.0.37, 2018-04-14) -> `serde-1.0.136` [rylev, [#1233](https://github.com/rust-lang/rustc-perf/pull/1233), [#1244](https://github.com/rust-lang/rustc-perf/pull/1244)]
- [x] `stm32f4` (0.12.1, 2021-03-07) -> `stm32f4-0.13.0` \[Kobzol, [#1241](https://github.com/rust-lang/rustc-perf/pull/1241)\, [#1250](https://github.com/rust-lang/rustc-perf/pull/1250)]
- [x] `syn`* (0.11.11, 2017-11-02) -> `syn-1.0.89` [rylev, [#1240](https://github.com/rust-lang/rustc-perf/pull/1240)]
- Plus keep the old one as part of the stable set. [[#1253](https://github.com/rust-lang/rustc-perf/pull/1253)]
- [x] `unicode_normalization` (0.1.7, 2019-01-15) -> `unicode-normalization-0.1.19` \[Kobzol, [#1243](https://github.com/rust-lang/rustc-perf/pull/1243), [#1249](https://github.com/rust-lang/rustc-perf/pull/1249)\]
- Note the hyphen rather than underscore in the name :smile:

Ones that require decisions.

- [x] `encoding`* (0.3.0-dev, pre-2017-11-02) -> demote to stable
- Hasn't been updated in 5 years. Has some big tables, but so do `ucd` and `tuple-stress`, and `unicode_normalization` also has some similarities, so not that interesting. Demote to the stable set. [nnethercote, [#1222](https://github.com/rust-lang/rustc-perf/pull/1222)]
- https://github.com/hsivonen/encoding_rs is an interesting alternative, could go in the primary set? But we do already have `unicode_normalization`...
- [x] `html5ever`* (0.5.4, pre-2017-11-02) -> `html5ever-0.26.0`
- Still interesting, especially for macro expansion. Actively developed. Latest release on crates.io was 2.5 years old, until Josh Matthews did 0.26.0 on request. Keep the old version for the stable set. [nnethercote, [#1257](https://github.com/rust-lang/rustc-perf/pull/1257)]
- [x] `inflate`* (0.1.0, pre-2017-11-02) -> demote to stable
- Latest release on crates.io is 3 years old. Profiles often look like a less extreme version of `keccak`, which we are keeping. Demote to the stable set. [nnethercote, [#1219](https://github.com/rust-lang/rustc-perf/pull/1219)]
- [x] `style-servo`* (0.0.1, pre-2017-11-02) -> demote to stable
- Biggest benchmark in the suite, not sure it carries its weight. Not as important a part of the Rust ecosystem as it once was. Also not released as a standalone crate. The code in Firefox has now diverged significantly from that in Servo. Demote to stable-only. \[nnethercote, [#1206](https://github.com/rust-lang/rustc-perf/pull/1206)\]
- [x] `tokio-webpush-simple`* (0.1.0, pre-2017-09-02) -> demote to stable
- Not released as a standalone crate. Very old style of async code. Demote to stable set. \[nnethercote, [#1220](https://github.com/rust-lang/rustc-perf/pull/1220)\]
- [x] `ucd` (0.1.1, 2018-08-24) -> move to secondary~~
- Unchanged, no new version in five years, no dev work in that time, and it was added as a stress test. Move to secondary. \[nnethercote, [#1221](https://github.com/rust-lang/rustc-perf/pull/1221)\]
- [x] `webrender` (0.57.2, 2018-05-23) -> `webrender-2022`
- Latest release (0.61.0) is 2 years old, but the repository has active development. Import a dev version, call it `webrender-2022` (nicer to type and view than `webrender-$revision`). \[nnethercote, [#1236](https://github.com/rust-lang/rustc-perf/pull/1236), [#1242](https://github.com/rust-lang/rustc-perf/pull/1242), [#1252](https://github.com/rust-lang/rustc-perf/pull/1252), [#1256](https://github.com/rust-lang/rustc-perf/pull/1256)\]
- [x] `webrender-wrench` (0.3.0, 2020-01-17) -> remove
- Much the same story as webrender. Do we really need this? I don't feel like it is that interesting. The collector README says "An executable pulling in large dependencies" but we don't measure compilation of dependencies. And it's incredibly slow to compile, even though the final crate is quite small, which is a bad trade-off in terms of CI usage. \[nnethercote, [#1237](https://github.com/rust-lang/rustc-perf/pull/1237)\]

## Secondary

Many of these will be unchanged.

- [x] `await-call-tree`
- [x] `coercions`
- [x] `ctfe-stress-4` [rylev, [#1286](https://github.com/rust-lang/rustc-perf/pull/1286), [#1288](https://github.com/rust-lang/rustc-perf/pull/1288)]
- Shrink this so it's not so long-running, rename it `ctfe-stress-5`
- [x] `deeply-nested`, `deeply-nested-async`, `deeply-nested-closures`
- \[nnethercote, [#1205](https://github.com/rust-lang/rustc-perf/pull/1205), [#1210](https://github.com/rust-lang/rustc-perf/pull/1210)\] Combine into `deeply-nested-multi`
- [x] `deep-vector`
- [x] `derive`
- [x] `externs`
- [x] `issue-46449`
- [x] `issue-58319`
- [x] `issue-88862`
- [x] `keccak` (0.1.0, 2018-07-30) -> `keccak-0.1.0`?
- No new release in four years. Code in the repo is a little different to our copy, less aggressive with unrolling, which results in a smaller `f1600` function and less stress on the compiler. Some very recent dev work started after a four year break. Leave it as a secondary benchmark.
- [x] `many-assoc-items`
- [x] `match-stress-enum`, `match-stress-exhaustive_patterns`
- \[nnethercote, [#1208](https://github.com/rust-lang/rustc-perf/pull/1208), [#1216](https://github.com/rust-lang/rustc-perf/pull/1216)\] Combine into `match-stress`
- [x] `projection-caching`
- [x] `regression-31157`
- [x] `token-stream-stress`
- [x] `tuple-stress`
- [x] `unify-linearly`
- [x] `unused-warnings`
- [x] `wf-projection-stress-65510`
- [x] `wg-grammar`

## Stable

Keep these ones around, unchanged, but only for the [dashboard](https://perf.rust-lang.org/dashboard.html)? I.e. don't run them as part of the normal benchmarking runs, and don't bother collecting the "master" data point any more.

## To add

To add?
- [x] Popular crates
- [x] `serde_derive-1.0.136` \[nnethercote, [#1306](https://github.com/rust-lang/rustc-perf/pull/1306)\]
- [x] `libc-0.2.121` \[nnethercote, [#1313](https://github.com/rust-lang/rustc-perf/pull/1313)\]
- ~~`proc-macro2-1.0.36`, `quote-1.0.17`, `log-0.4.16`, `memchr-2.4.1`, `tokio-1.17.0`~~
- Decided against, the profiles aren't that interesting, similar to existing benchmarks
- ~~Build scripts? e.g. `libc`, `proc-macro2`, `syn` are popular ones. Turn out to not be interesting, though, just look like any small program.~~
- [x] Interesting profiles
- [x] Macro expansion: `tt-muncher` \[nnethercote, [#1290](https://github.com/rust-lang/rustc-perf/pull/1290)\]
- Chosen ahead of of `async-std-1.10.0`, `time-macros-0.2.3`, `yansi-0.5.0`, and several other crates, because it's a purer stress test, which is useful.
- [x] `bitmaps-3.1.0` \[nnethercote, [#1314](https://github.com/rust-lang/rustc-perf/pull/1314)\]
- stresses trait stuff, similar to `nalgebra-0.30.1`
- ~~`http-0.2.6`: has a very large function, also pretty popular~~
- Similar to `keccak`, but less extreme
- ~~`nalgebra-0.30.1`~~
- `bitmaps-3.1.0` has similarities, but is smaller and more extreme, and therefore a better choice
- ~~`wast-39.0.0`: obligations processing, esp. if we remove `inflate` and/or `keccak`~~
- We kept `keccak`
- ~~`web-sys-0.3.56`: parsing heavy if no features are enabled; proc-macro heavy if many features are enabled~~
- A bit too weird for now

# Instructions

See https://github.com/rust-lang/rustc-perf/pull/1217 for an example of one that's already been done.

See https://github.com/rust-lang/rustc-perf/pull/1295 for detailed instructions.