Steering Meeting info

Title: Discuss results of 2020 Contributor Survey

Estimate: Maybe 2, there is a lot here.
Type: non-technical (mostly)

Summary

There were 409 responses.

• 164 were people who have never had a PR merged
• 169 contributed <1PR/month
• 76 contributed >=1PR/month

We break down the responses by these 3 categories of contributors, making specific observations and conclusions. The observations/conclusions here are all by mark-i-m, so please take with a grain of salt…

Never had PR merged:

• Need clear entry points, streamlined contribution pathway
• Need better ways to find work
• Better organization of information

<1PR/month:

• Some want to make drive-by contributions, but this is hard with slow compile times
• Some want to get more involved but don't know how
• Compile times, knowledge of codebase are major barriers
• Need more easy/mentored issues
• Perhaps point to other parts of project (not rust-lang/rust)
• Other parts of project seem understaffed, PRs ignored
• Interactions with team members have high impact for new contributors

>=1PR/month:

• Work done to help new contributors will also help frequent contributors
• Improvements to compile time, rustc-dev-guide seem to be even more helpful for this group than other
• Better organization of information would help
• Lots of papercuts

Motivation

We wanted to find out the pain points faced by contributors. We conducted the survey, and these are the results.

Details

Details are below in this document.

Challenges

• There is a lot of feedback here. Processing it might take time.
• The high-impact items are hard (e.g. compile time) or require dedicated time by experts (e.g. mentoring, documentation).
• There are also a lot of smaller papercuts and hinderances too. It's not clear how to address those in a time efficient way.

Key design questions

The key questions to be answered by the meeting are:

• What concrete steps/action items should be taken to improve contributor experience?
• How should the above plan relate to parts of the rust project that are not in rust-lang/rust?

Survey Notes

Without further ado, here are my notes on the survey results…

Never had a PR merged

There were 164 out of 409 respondents in this category.

The overwhelming majority of these respondents had never submitted a PR. Some had started but never finished working on a PR. Only 4 submitted a PR that was rejected.

Takeaway 0

Observations:

• CONTRIBUTING.md is most commonly cited resource, followed by README
  • These were the only resources used by more than half of respondents and then only by 63%
• Less than half of new contributors used rustc-dev-guide or any of the other resources

Conclusions:

• We need to give clear guidance to new contributors on where to look for information (i.e. probably rustc-dev-guide)
• We need to have a single clearly highlighted resource that is the canonical starting point

Unresolved questions:

• The above currently doesn't break down results by what people tried to contribute to. If we filter out those who were not contributing to rust-lang/rust, are the results the same?

Takeaway 1

Observations:

• Most new contributors are interested in the most user-visible aspects of rust: the compiler, standard libs, and tooling
• Interest in writing docs is surprisingly high
• There is non-trivial interest in contributing to other things (not the above)

Conclusions:

• Streamlining contributions to the compiler, standard libs, and tools should be a priority; otherwise, we are losing a lot of potential help

Unresolved Questions:

• Are there different pain-points for those interested in contributing to tooling as opposed to the compiler/standard libs?

Takeaway 2

Observations:

• Almost 1/2 of respondents in this category couldn't find a suitable issue to work on
• More than 1/3 reported that the codebase was too hard to learn
• About 1/5 reported not feeling comfortable contributing to code or discussion
  • Many people also noted lack of knowledge, confidence, experience, or not knowing where to start
• New contributors don't run into issues related to building, testing, and reviewing as often, but some do

Conclusions:

• We need to clearly identify and point to ways new contributors can get involved with low initial committment
• We need to have a clear "Getting Started" page somewhere

Unresolved Questions:

• Is the lack of entry-level issues just due to the complex nature of the project? Or is there something we can do about it?

Takeaway 3

Observations:

• A bit more than 1/3 of respondents stated that "people were helpful" (more than in other communities)

Conclusions:

• We should continue to do a good job here

Unresolved Questions:

• We should try understand where we are not doing a good job here. For example, I have an impression that contributions to some parts of the project tend to be ignored due to lack of manpower.

Takeaway 4

Observations:

• The free form comments seem to be all over the place, but there are a few common themes
  • It's hard to tell the status of issues, features, and the project in general. Information is scattered and you have to be an "insider" to know what's going on.
  • There is a lot of missing documentation, and this can be a hinderance
  • It's hard to find good starting issues to work on, but contributing to documentation might be a good entry point.
  • Lots of people seem to have an emotional reaction (something like intimidation/discouragement) when daunted by the size of the rust project
    • One respondent: "Maybe I'm not enough to contribute to Rust…"
  • Lots of people are having a hard time with various aspects of tooling, including bootstrapping, slowness of compile times, slow IDE experience, etc. These make it hard to make an initial contribution and are discouraging.
  • A handful of people have had negative interactions with team members, including negative interactions on RFCs, lack of PR reviews, or rejection of contributions.
  • A handful of contributors have hinderances that I don't think we can solve, like not knowing rust well enough or not liking github.

Conclusions:

• We should work on organization of information.
• We should work to remove the intimidation/make contributing less discouraging. It should not feel like it requires superhuman intellegence or insider knowledge.
• (rehash from Takeaway 2) We need to lower the initial time investment from reading about the compiler or bootstrapping because these are barrier that discourage people from contributing in the first place.

Contribute < 1/month

There were 169 out of 409 respondents in this category.

Takeaway 5

Observations:

• Some respondents are more interested in quick drive-by contributions that fill a couple of hours of down time on some day
• Compile times/complex setup are a major barrier to drive-by contributions.

Conclusions:

• We should have a good way for people to make drive-by contributions without getting more involved
• We should create clear guidelines and documentation on how to build/test rust-lang/rust faster
• It might make sense to push drive-by contributors towards using ./x.py check + CI, rather than building locally

Takeaway 6

Observations:

• Some respondents are interested in getting more involved in contributing to rust, but are facing barriers to do so.
• Many respondents noted that
  • It was difficult to see what the current status of things was without actively following communication channels (e.g. zulip, discord)
  • GH issues often are claimed but then it's later not clear what their status is
  • It's not clear where work is needed or how to start
• One person also noted that it's not clear how to join teams or WGs

Conclusions:

• We should also have a smooth, well-documented path for those who do want to get more involved
• We should document expectations for team members/contributors (I think already partially done), such as review responsibilities, contribution regularity, etc…
• We should broadly link to this documentation for both teams and WG from multiple places (e.g. the rust website and the rustc-dev-guide)

Takeaway 7

Observations:

• One of the most commonly mentioned things both in the checkboxes and the free-form answers was the difficulty of finding things to work on
• Mentored issues or even quick hints on the GH issues are extremely helpful
• Easy/beginner/mentored issues tend to be grabbed very quickly
• It's often hard to figure out what the status of an issue is later (e.g. is it still being worked on by someone who claimed it earlier)

Conclusions:

• We need to be proactive about identifying easy/mentored issues. I don't think a WG can do this; compiler-team folks will have to pitch in here.
• We need to triage E-easy/E-mentor issues frequently and make sure the status is clear and fairly fresh

Unresolved Questions:

• What else can we do here? Is there actually more beginner-level work? Or should we nudge beginners elsewhere?

Takeaway 8

Observations:

• The pain points "too slow to build" and "frustrating to rebuild after small changes" shot to the top of the list, unlike the previous group
• It seems that one of the first things people run into is the complexity/slowness of setting up building the compiler.
• A couple of people requested "cookbook-like" resources for how to make contributions/what commands to run.

Conclusions:

• We should prioritize solving this problem. It seems to be a big blocker for many people.
• We should push on initiatives that improve compile time for rust-lang/rust in particular
• We should make it easier to compile only some parts of the project (not the whole)
  • Librarification might be very helpful for getting people involved
  • We should also persue making std libs, rustdoc, etc easier to compile on their own
• We should prioritize the rustc-dev-guide "Getting Started" guide and make sure that experts/frequent contributors input their tips.

Takeaway 9

Observations:

• Most contribution attempts were to the std libs, compiler, documentation, or dev-tools (cargo, clippy, etc). There is a large gap to the next thing.
• By far the most common pain points mentioned (especially in the free-form comments) were:
  • Pain frustration with building (too slow, too much disk space, x.py confusing, bootstrapping confusing, etc)
  • Not being able to find suitable things to work on; not being able to tell the status of things
  • Having a hard time to learn the codebase
• Some noted that working on chalk/polonius/etc was easier because the build/test time was shorter

Conclusions:

• All of the above takeaways… but also…
• We should evaluate whether more easy/mentored issues can be created or if we should push interested people towards WGs or something?
• Adjacent compiler efforts (e.g. chalk, polonius) might be good way to start getting more involved
• We should publicize opportunities on non-rust-lang/rust more widely

Takeaway 10

Observations:

• Multiple people cited PRs that they opened which were ignored for weeks/months and/or bitrotted
• This is very discouraging, especially to newcommers
• This tends to happen on parts of the project that feel "less glamorous", including the error docs, crates.io, docs.rs, and cargo
• Some also cited RFCs that they have opened but which have been largely ignored for years

Conclusions:

• Work on these parts of the project seem very understaffed, especially in reviewers/experts. It feels like they are in maintainence mode.
• Perhaps we should be concerned about this?
• We should have a light-weight process to green-light opening an RFC at all (e.g. MCPs)

Unresolved questions:

• What even is a solution here?
• Should we discourage people from making contributions or have explicit warnings of long review times/maintainence mode?

Takeaway 11

Observations:

• A few people noted various negative experience with respect to teams:
  • One person accidentally made a merge commit (and apparently has very little knowledge of git). It seems that the reviewer's response scared the person off and discouraged them.
  • One person did a refactoring at the suggestion of one team member only to be told by the reviewer that it made the code worse.
• One person felt that asking team members for help was wasting busy expert time
• One person said: "The Rust team does a really good job of leading you in the right direction, supporting you, and making you feel welcome."

Conclusions:

• We need to remind team members to be patient (maybe even long-suffering), especially with unfamiliar contributors, to leave a good first impression.
• We need to be careful of burning out/exhausting team members so that they can be patient/gracious with others. Some team members have themselves mentioned burnout and organizational debt issues that are wearing on them.

Takeaway 12

Observations:

• Most people seem to feel that "people are helpful"
• Half felt that the rust project made them feel "comfortable contributing to code or discussions"
• Some felt that discussion are "unproductive or exhausting"

Conclusions:

• We are doing well so far, but we could do better.
• As the community grows, it will be increasingly important to control discussions.

Contribute >= 1/month

There were 76 out of 409 respondents in this category.

Takeaway 13

Observations:

• Among frequent contributors, there are a few notable difference from the previous groups:
  • The compiler is by far the most frequently contributed-to codebase, 7/10 reporting that they contributed to the compiler
  • About half report contributing to the std libs or dev-tools
  • reading the code is the most frequent resource, followed by the rustc-dev-guide
  • ~3x more likely to use zulip
• Interestingly, more than half still report using CONTRIBUTING.md or READMEs in the repo

Conclusions:

• Easing pain points w.r.t. to the compiler will help frequent contributors more than improving other parts of the project.
• ecstatic-morse's idea of integrating CONTRIBUTING.md into rustc-dev-guide seems to be a good one.

Unresolved question:

• Should confirm that CONTRIBUTING/READMEs apply for those who contributed to rust-lang/rust, and not just conflating with other repos…

Takeaway 14

Observations:

• The top pain points are mostly related to velocity
  • Frustrating to rebuild after small changes
  • Building the compiler or LLVM takes too long (more people care about the compiler, though)
  • Review times too long
  • Disk space usage
• Interestingly, 3/10 reported that the codebase is too hard to learn
• About 1/5 report not finding suitable issues
• "Better than other projects": 7/10 report that others were helpful; >6/10 report feeling comfortable

Conclusions:

• Working on compile time will make a huge difference to frequent compiler devs. This is one of the biggest things people mentioned.
• Improving the rustc-dev-guide helps frequent contributors the most.
• We've done a good job serving frequent contributors.

Unresolved questions:

• Somehow many frequent contributors "have a hard time finding suitable issues"… perhaps we need to push harder on WGs?

Takeaway 15

Observations:

• Contributors to compiler, stdlib report getting timely feedback
• Contributors to other parts of the project (e.g. rustdoc, docs.rs, crates.io, other crates) often see long delays
• rust-lang/rust stuff takes a long time to compile; it seems like almost everyone runs into this problem
• Contributors to other parts of the project report that faster compile times are really helpful

Conclusions:

• Contributor experience is very inconsistent across the project, with some parts having slow compile times and others having slow review times.
  • It's not clear how much either of these problems can be solved…
• We need more manpower for the many crates/subprojects that are not rust-lang/rust

Takeaway 16

Observations:

• There are a variety of unusual pain points/papercuts mentioned by frequent contributors that are not mentioned by other groups
• There are also a bunch of pain points that are common to all of the groups of contributors

Conclusions:

• Work done to improve things for new contributors will also help frequent contributors
• We need some way to keep track of all the little things that frequent contributors run into and try to address them

Takeaways 17

Observations:

• There were a bunch of free-form pain points that would be solved by different forms of documentation everywhere:
  • PRs often lack context, why created?
  • Review comments can be inaccessible to new contributors, lack context
  • Documentation exists mainly in experts' heads
  • x.py poorly documented, but easy to iterate b/c python
  • Tutorials about how to do things could be helpful (e.g. contribute a bug fix, small feature, rfc)
  • "The project perhaps needs a more established technical roadmap."
  • During large/incremental refactorings, it can be hard to tell what is the right/newer way to do stuff
  • Poor commit messages make it harder to understand why things are the way they are
  • Lack of comments in rustc
• There were also a bunch of miscillaneous pain points and papercuts that might be worked around if we wanted to and had the man power (e.g. git submodule pain, "offline mode" compiler builds, reapproval after rebasing, etc)

Conclusions:

• We should consider prioritizing documentation in some of the above areas, despite the extra work that devs often don't enjoy.
• We should maybe keep a list of these things. Perhaps they can be good introductory issues?

General Thoughts (from mark-i-m)

• Sense "lostness" may be due to a mix of actual complexity, perceived complexity, poor/scattered documentation, and failure to do "obviously simple" things like building, perceived rapid pace of change
• A few people commented that things have improved
  • compile times since 1.0 (incremental, parallel codegen)
  • code structure, guidelines, WGs, and rustc-dev-guide make it less scary
• One person commented that quick drive-by starting-point comments from experts are helpful
• We need a few entry-level ways to contribute. Writing compiler code is almost certainly not going to be it because learning how the compiler works is hard and overwhelming for most people. Some ideas:
  • Writing doc comments (e.g. this is how I learned how the MBE system works). Perhaps we can make this a formal WG?
  • ICE-breakers
  • Perhaps scribing for (public) team/WG meetings? Not sure how useful this task actually is…
  • Contributing to not-rust-lang/rust, though this won't teach you how to the compiler works
• Bad combination: most people who stopped did so due to time constraints, but a lot of the pain points people have are about things taking too long
• Contributing to the standard library independently of the compiler should become the standard (no pun intended) mode of operation
• We should make sure people understand why the compiler, std libs, etc are built the way they are; otherwise, it feels opaque and needlessly complicated.
• More experienced users seem to be more prone to work on projects outside the rust-lang/rust repo
  • Perhaps we can highlight these more as ways for new people to contribute?
• When people make mistakes, especially newcomers, the response needs to be gentle and patient. Otherwise, we scare people off or discourage them.
• The bootstrapping process seems to come up as a common source of confusion, especially the stage naming. Might good to rename things so that there is an easy rule to understand why things are the way they are, and also so that people can tell more easily how much they need to build for some contribution.
• Might make sense to point people to ./x.py check and encourage people to use CI rather than just building locally…
• Various other pain points mentioned by people
  • One person noted that they don't know how to contribute anonymously. I found this interesting because I myself largely gave up. It's too easy to accidentally leave a school email address in a git commit…
  • One person noted that it would be nice to have an "offline" mode – a way to build that doesn't require network
  • Two people mentioned the need for python to build. This is apparently more painful on windows.
  • A handful of people mentioned that cargo build and cargo test don't work, confusingly.
  • One person: no way to retry ci after spurious failure
  • One person: PRs often lack context or explanation; likewise with review comments
  • etc
• Idea: would it be feasible to have a small build farm using bors? Provide a build that people can play with/debug/test?