---
# System prepended metadata

title: perf.rlo
tags: [' rustc', perf.rlo]

---

---
tags: perf.rlo, rustc
---

# perf.rlo

## What are goals of meeting?

 * Use meeting time itself to review site and/or establish goals?

 * What action items can we establish?

 * Who are the stake-holders?
     * Do we think we have sufficient present here to assign action items usefully?
     * Do we think we have sufficient present here to make decisions?

## Topics
 * Who are the groups of people that consume this data and what do each of these groups want to see when visiting the site?
     * Compiler team: curious as to the general trajectory of performance
     * Those actively working on performance (perf team or motivated contributor)
     * rustc dev tracking performance impact of their changes
     * Curious non-dev person looking to see whether rustc is fast enough
     * Curious person who uses Rust
 * Who should establish the set of benchmarks?
     * How often should the set change?
     * How can we be confident this captures a reasonable slice of "representative" Rust programs?

 * Do we need to gather expertise in Data Science
     * Is there research-worthy material here? simulacrum thinks we are running up against open research problems (at least not with well known solutions)

 * Changes/enhancements to discuss
     * How to present the metrics we gather today
         * Do the current defaults reflect what matters?
         * Cannot currently visualize more than one metric at once. 
         * Should we change how the metrics are stored?
     * What metrics to gather in the future
         * Disk space usage is an obvious one we need
         * pnkfelix thinks more stuff about cache-misses also needed (more real-world impact than instruction counts)
         * (Note that perf counters have architectural limits)
     * How do we track cross platform performance?
     * What automation could be added, to ease existing workflows (e.g., triage report)?

 * "Competitors" exist
    * e.g. arewefastyet.rs
    * should we embrace these (e.g. actively advertise them), leverage them as staging grounds for ideas?
        * That is, ideas w.r.t what to benchmark, what metrics to gather, and how to present them.

# collected pain points

* Not clear which benchmarks matter. Am I supposed to weight all equally? (Does summary weight all equally?)
* noise, in particular on some benchmarks, is frequently present, and makes it easy to essentially ignore such benchmarks.
* the compare mode presentation, for each benchmark, has one "full" and then several incr variants. This might be leading to over-emphasis on incr performance.
* How do I tell what things are likely noise vs real gains and losses in performance
* too much data -- boiling data down to a view of "how is compiler doing". The dashboard helps here, but is it representative?
* regressions have unclear responsibility in terms of tracking down cause/fixing and deciding whether to revert. (is it the author? performance triager?)
* regressions during a rollup have unclear provenance
* hard to get a good overview of performance (of a particular diff) across many metric types (e.g.., wall time, cpu instructions, rss, etc.)
* who decides how much regression is "too much"?
* The detailed comparison view often does not correlate with summarized comparison view for small regressions/speedups. 
* making decisions based on compare view often seems fairly arbitrary (eh, some minor regressions, some minor improvements, seems good enough :shrug:)
* We don't have disk space as metric. :)
* "well known" that some components are causing major increases in timing (e.g., simd in core/std), but unclear if we can say "stop adding intrinsics" or how to evaluate the tradeoff between features and performance. particularly performance of rustc developers vs end users, too
* The impact of cgu partitioning is not controlled for, while introducing systematic bias in measurements.
* overwhelming for newcomers on how to understand what they're looking at
* no measurement of runtime performance of generated code (kind of orthogonal, kind of not)

## Understanding the issue

With a focus on rustc devs who use perf.rlo to gauge the performance impact of their changes to the compiler, it's important to understand a typical workflow:

* Dev introduces change
* Dev or a reviewer kicks off a perf run to see what the impact is
* TODO FINSIH