Rust Metrics Planning Document

Context
Aims
- Trust: Do not violate trust of our users
  - NO TELEMETRY, NO NETWORK CONNECTIONS
  - emit metrics locally
  - user information should never leave their machine in an automated manner, sharing their metrics should always be opt in, clear, and manual
  - All of this information would only be stored to disk, with some minimal retention policy to avoid wasteful use of users’ hard drives
- feedback: improving feedback loops to assist with iterative improvement within the project
  - answer questions from real production environments in a privacy preserving way
  - improve legibility of rare or intermittent issues
  - earlier warnings for ICEs and other major issues on nightly, improving likelihood that we'd catch the issues before they hit stable.
    - https://blog.rust-lang.org/2021/05/10/Rust-1.52.1.html
- performance impact: leave no trace (minimize performance impact, particularly for default enabled metrics)
- Extensible: it should be easy to add new metrics as needed
  - Only add metrics as a way to answer a specific question in mind, with an explicit documented rationale
- user experience: improving user experience of reporting issues to the project
Quests (provisional)
- Supporting rust feature development
  - Improved ICE Reporting
  - build tooling for automated cleanup of metrics in cargo
  - collate information in crater
  - build support for reporting specific issues back to relevant contributors
    - e.g. cargo plugins, vscode, or rust-analyzer could be used to locally analyze metrics for issues rust teams care about and present users with an appropriate summary to file as a ticket
- Direct improvements of User Experience e.g. identifying which compiler errors to focus on improving
- Improve perf feedback loops and insight
  - collate information in perf

2024/08/08 10:11:37

manual

manual as default is good. But I think an explicit `--yes-please-prompt-me-occasionally-to-feed-the-data-kraken` could be ok for those who want to provide metrics regularly.

Wesley Wiser

2024/08/15 16:07:08

emit metrics locally

It might improve user trust if there were two steps here: one to enable metric collection locally (disabled by default) and then a second step to actually send the metrics off the machine.

2024/08/15 16:08:06

clear

Perhaps this is what's meant by "clear" but I think it's necessary that the metric submission format be in plain text so that it's easy for users to see exactly what data is being sent.

2024/08/15 16:13:10

privacy preserving

I think this is really important, thank you for calling it out! In general, I would suggest we strongly discourage collection of arbitrary "text" via metrics. Collecting text makes it very easy to accidentally obtain user's names, project code or other sensitive identifiers. Collecting ICE stack traces is really important and users generally seem to be comfortable sharing that when reporting issues so I think that is an important *exception* and should be documented as such.

2024/08/15 16:15:16

Trust

Go's telemetry is publicly available (https://telemetry.go.dev/). I think making our telemetry data public as well would be another step to improve trust since there is no "hidden" data we're keeping for ourselves.

2024/08/15 16:16:39

Extensible

It would be great if this could be a drop-in library for all kinds of Rust tools. I imagine Cargo and r-a have things they'd like to have metrics on as well so making it easy to re-use infrastructure here would be a big win!

Rust Metrics Planning Document

Context

Aims

Quests (provisional)