rustc-perf benchmarks upgrade 2022

As per the roadmap, we want to improve the rustc-perf benchmark suite.

Task completion is indicated like so:

  • incomplete
  • complete

As of 2022-04-29, the upgrade is complete!

Notes:

  • Real-world crates include the version number in their name, for clarity, and to allow for future updates.
  • No pre-release versions, just proper release, to avoid annoying version numbers like 1.2.3-pre.1.
  • For dates below: there was a big benchmarks rearrangement on 2017-11-{02,03}, and getting version history before then is a pain, so I haven't gone back farther than that. That's plenty old.
  • New benchmarks should be given .patch files to test incr-patched.

Primary

Ones marked with * are part of the "stable" set (see below).

Easy changes. (Latest version numbers were correct at time of writing, but may have changed.)

See the instructions at the bottom of this document for the update procedure.

  • cargo (0.29, 2018-05-23) -> cargo-0.60.0 [nnethercote, #1209, #1211]
  • clap-rs (2.29.0, 2018-02-25) -> clap-3.1.6 [nnethercote, #1217, #1235]
  • cranelift-codegen (0.30.0, 2019-04-11) -> cranelift-codegen-0.82.1 [lqd, #1228, #1247]
  • diesel (2.0.0??, 2020-12-03) -> diesel-1.4.8
    • Version number in collector/benchmarks/diesel/diesel/Cargo.toml says 2.0.0, weird [lqd, #1238, #1248]
  • futures* (0.1.0, pre-2017-11-03) -> futures-0.3.21
    • The futures-0.3.21 crate is now just a facade crate with a bunch of sub-crates: futures-core, futures-i/o, etc. Because we only measure leaf crate compilation, it's boring and not that different to helloworld. So we can just move the existing futures crate (which does have actual code in it) to the stable set. [nnethercote, #1218]
  • helloworld (unchanged)
  • hyper-2 (0.13.0-alpha.4, 2019-11-20) -> hyper-0.14.18 [lqd, #1251, #1262]
  • piston-image* (0.10.3, pre-2017-11-02) -> image-0.24.1 [rylev, #1229]
    • Plus keep the old one as part of the stable set. (#1234)
  • regex* (0.1.80, pre-2017-11-02) -> regex-1.5.4 [rylev, #1231, #1239]
    • Plus keep the old one as part of the stable set.
  • ripgrep (0.8.1, 2018-05-23) -> ripgrep-13.0.0 [Kobzol, #1224, #1232]
  • serde (1.0.37, 2018-04-14) -> serde-1.0.136 [rylev, #1233, #1244]
  • stm32f4 (0.12.1, 2021-03-07) -> stm32f4-0.13.0 [Kobzol, #1241, #1250]
  • syn* (0.11.11, 2017-11-02) -> syn-1.0.89 [rylev, #1240]
    • Plus keep the old one as part of the stable set. [#1253]
  • unicode_normalization (0.1.7, 2019-01-15) -> unicode-normalization-0.1.19 [Kobzol, #1243, #1249]
    • Note the hyphen rather than underscore in the name
      Image Not Showing Possible Reasons
      • The image file may be corrupted
      • The server hosting the image is unavailable
      • The image path is incorrect
      • The image format is not supported
      Learn More →

Ones that require decisions.

  • encoding* (0.3.0-dev, pre-2017-11-02) -> demote to stable
    • Hasn't been updated in 5 years. Has some big tables, but so do ucd and tuple-stress, and unicode_normalization also has some similarities, so not that interesting. Demote to the stable set. [nnethercote, #1222]
    • https://github.com/hsivonen/encoding_rs is an interesting alternative, could go in the primary set? But we do already have unicode_normalization…
  • html5ever* (0.5.4, pre-2017-11-02) -> html5ever-0.26.0
    • Still interesting, especially for macro expansion. Actively developed. Latest release on crates.io was 2.5 years old, until Josh Matthews did 0.26.0 on request. Keep the old version for the stable set. [nnethercote, #1257]
  • inflate* (0.1.0, pre-2017-11-02) -> demote to stable
    • Latest release on crates.io is 3 years old. Profiles often look like a less extreme version of keccak, which we are keeping. Demote to the stable set. [nnethercote, #1219]
  • style-servo* (0.0.1, pre-2017-11-02) -> demote to stable
    • Biggest benchmark in the suite, not sure it carries its weight. Not as important a part of the Rust ecosystem as it once was. Also not released as a standalone crate. The code in Firefox has now diverged significantly from that in Servo. Demote to stable-only. [nnethercote, #1206]
  • tokio-webpush-simple* (0.1.0, pre-2017-09-02) -> demote to stable
    • Not released as a standalone crate. Very old style of async code. Demote to stable set. [nnethercote, #1220]
  • ucd (0.1.1, 2018-08-24) -> move to secondary~~
    • Unchanged, no new version in five years, no dev work in that time, and it was added as a stress test. Move to secondary. [nnethercote, #1221]
  • webrender (0.57.2, 2018-05-23) -> webrender-2022
    • Latest release (0.61.0) is 2 years old, but the repository has active development. Import a dev version, call it webrender-2022 (nicer to type and view than webrender-$revision). [nnethercote, #1236, #1242, #1252, #1256]
  • webrender-wrench (0.3.0, 2020-01-17) -> remove
    • Much the same story as webrender. Do we really need this? I don't feel like it is that interesting. The collector README says "An executable pulling in large dependencies" but we don't measure compilation of dependencies. And it's incredibly slow to compile, even though the final crate is quite small, which is a bad trade-off in terms of CI usage. [nnethercote, #1237]

Secondary

Many of these will be unchanged.

  • await-call-tree
  • coercions
  • ctfe-stress-4 [rylev, #1286, #1288]
    • Shrink this so it's not so long-running, rename it ctfe-stress-5
  • deeply-nested, deeply-nested-async, deeply-nested-closures
    • [nnethercote, #1205, #1210] Combine into deeply-nested-multi
  • deep-vector
  • derive
  • externs
  • issue-46449
  • issue-58319
  • issue-88862
  • keccak (0.1.0, 2018-07-30) -> keccak-0.1.0?
    • No new release in four years. Code in the repo is a little different to our copy, less aggressive with unrolling, which results in a smaller f1600 function and less stress on the compiler. Some very recent dev work started after a four year break. Leave it as a secondary benchmark.
  • many-assoc-items
  • match-stress-enum, match-stress-exhaustive_patterns
    • [nnethercote, #1208, #1216] Combine into match-stress
  • projection-caching
  • regression-31157
  • token-stream-stress
  • tuple-stress
  • unify-linearly
  • unused-warnings
  • wf-projection-stress-65510
  • wg-grammar

Stable

Keep these ones around, unchanged, but only for the dashboard? I.e. don't run them as part of the normal benchmarking runs, and don't bother collecting the "master" data point any more.

To add

To add?

  • Popular crates
    • serde_derive-1.0.136 [nnethercote, #1306]
    • libc-0.2.121 [nnethercote, #1313]
    • proc-macro2-1.0.36, quote-1.0.17, log-0.4.16, memchr-2.4.1, tokio-1.17.0
      • Decided against, the profiles aren't that interesting, similar to existing benchmarks
    • Build scripts? e.g. libc, proc-macro2, syn are popular ones. Turn out to not be interesting, though, just look like any small program.
  • Interesting profiles
    • Macro expansion: tt-muncher [nnethercote, #1290]
      • Chosen ahead of of async-std-1.10.0, time-macros-0.2.3, yansi-0.5.0, and several other crates, because it's a purer stress test, which is useful.
    • bitmaps-3.1.0 [nnethercote, #1314]
      • stresses trait stuff, similar to nalgebra-0.30.1
    • http-0.2.6: has a very large function, also pretty popular
      • Similar to keccak, but less extreme
    • nalgebra-0.30.1
      • bitmaps-3.1.0 has similarities, but is smaller and more extreme, and therefore a better choice
    • wast-39.0.0: obligations processing, esp. if we remove inflate and/or keccak
      • We kept keccak
    • web-sys-0.3.56: parsing heavy if no features are enabled; proc-macro heavy if many features are enabled
      • A bit too weird for now

Instructions

See https://github.com/rust-lang/rustc-perf/pull/1217 for an example of one that's already been done.

See https://github.com/rust-lang/rustc-perf/pull/1295 for detailed instructions.

Select a repo