T-compiler Ambitions in 2022 (AWS copy)

--- title: T-compiler Ambitions in 2022 (AWS copy) breaks: false --- # T-compiler Ambitions in 2022 (AWS copy) *Editor's note (not part of blog post): Previously considered titles include: ~~T-compiler 2022 roadmap~~, ~~T-compiler astrological chart~~, ~~T-compiler horoscope~~, ~~T-compiler fortune cookies~~, ~~T-compiler crystal ball~~. Other suggestions welcome!* *Editor's note (not part of blog post): Our intent is to post this on the Inside Rust blog in the near future, circa February 18th. You can consider it an early preview; we want feedback from the members of T-compiler and T-compiler-contributors before we circulate it more broadly. You can post comments in the HackMD margin, or send us an email, or write in a [Zulip thread](https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/roadmap.20for.202022.20discussion/near/270258331).* Some people have been wondering about what the Rust Compiler Team has planned for 2022. This note is to let you all know what activities the team plans to focus on this year. This document is structured into three parts: our [Overall Themes][] for this year, the [Concrete Initiatives][] we have resources to drive, and [Aspirations][] for what we could do if given more help. [Overall Themes]: #Overall-Themes [Concrete Initiatives]: #Concrete-Initiatives [Aspirations]: #Aspirations [toc] ## Introduction Part of the motivation for this note is to encourage new contributors to get involved. We have a lot of newcomers, from individuals to large organizations, who are very excited about Rust's potential, and we want to show all of them what they can do to help. This is a list of items, divided into a [Concrete Initiatives][] section and an [Aspirations][] section. We accumulated these items during discussions with the Compiler Team and the Compiler Contributors. The [Concrete Initiatives][] have owners assigned; each has allocated time this year to attack the problem. The [Aspirations][], on the other hand, are items that the team agrees would be great areas for investment, but where we currently lack sufficient resources and experienced developers to make progress this year. This is *not* a list of everything we will do this year; at least, not without help. You can think of the [Aspirations][] part of the doc as an explicit call to arms: If you see something there that interests you, please reach out to the owners listed in that section to find out how you might be able to help. Furthermore, if you think we have left something important out of this list, please do reach out. Team leads [pnkfelix] and [wesleywiser] are happy to discuss what items are on this list, and what items we explicitly decided to leave *off* the list. [Aaron Hill]: https://github.com/Aaron1011  [bjorn3]: https://github.com/bjorn3  [cjgillot]: https://github.com/cjgillot  [davidtwco]: https://github.com/davidtwco  [estebank]: https://github.com/estebank  [lcnr]: https://github.com/lcnr  [mw]: https://github.com/michaelwoerister  [nikomatsakis]: https://github.com/nikomatsakis  [oli-obk]: https://github.com/oli-obk  [jackh726]: https://github.com/jackh726  [lqd]: https://github.com/lqd  [nnethercote]: https://github.com/nnethercote  [tmandry]: https://github.com/tmandry  [scottmcm]: https://github.com/scottmcm  [pnkfelix]: https://github.com/pnkfelix  [wesleywiser]: https://github.com/wesleywiser  [jswrenn]: https://github.com/jswrenn  [apiraino]: https://github.com/apiraino  ## Overall Themes [Developer Experience]: #Developer-Experience [Contributor Workflow]: #Contributor-Workflow [Delivering on Rust's Promise]: #Delivering-on-Rusts-Promise ### Delivering on Rust's Promise (🦀) Delivering on Rust's Promise is a cross-cutting theme; it means identifying the gaps between expectation and reality for each of our three pillars: [Performance, Reliability, and Productivity][rust-lang], and then addressing those gaps. [rust-lang]: https://www.rust-lang.org ### Developer Experience (👩‍💻) Improving on the Rust Developer Experience means asking and answering the question "what could we do to *delight* Rust developers?". We have opportunities to improve when developers are writing, when they are compiling, and when they are running Rust code. [Debugging concrete]: #Debugging-Initiatives [Debugging aspirational]: #Debugging-Aspirations [Debugging]: #Debugging-Object-Code [Faster concrete]: #Compilation-Time-Initiatives [Faster aspirational]: #Faster-Compilation-Aspirations [Expressiveness concrete]: #Language-Expressiveness-Initiatives [Expressiveness aspirational]: #Expressiveness-Aspirations ### Contributor Workflow (🛠️) Finally, improving the Compiler Contributor Workflow means technology enhancements that benefit people maintaining and extending the Rust compiler itself. (We also make non-technical enhancements, such as changes to our social processes, but this document focuses on technology.) [Librarification Initiatives]: #Librarification-Initiatives [Librarification Aspirations]: #Librarification-Aspirations [Compiler Backend]: #Compiler-Backend [Review Queue Crew]: #Review-Queue-Crew ## Concrete Initiatives This section is the closest thing to a "roadmap" we have for 2022. It is a list of important items with dedicated owners that have time allocated to make significant progress on the problem this year. ### I-unsound issues (🦀) As of this writing, we have 69 [open issues tagged I-unsound](https://github.com/rust-lang/rust/issues?q=is%3Aopen+is%3Aissue+label%3AI-unsound), and 44 of those are [also tagged T-compiler](https://github.com/rust-lang/rust/issues?q=is%3Aopen+is%3Aissue+label%3AI-unsound++label%3AT-compiler). In theory, any unsoundness issue potentially undermines Rust's promise of reliability. We want, by the end of this year, to have a clear understanding of how each of those I-unsound issues came to be. We are looking into systematically detecting such issues and whether we can deploy mitigations or fixes for entire classes of issues instead of addressing them on a case by case basis. [oli-obk], from Amazon Web Services, will be the primary owner of work in this space. Please reach out to [oli-obk] and [pnkfelix] (also from Amazon Web Services) if you are interested in helping resolve these issues! ### Async Rust Initiatives (🦀, 👩‍💻) There is significant overlap between async rust and other areas of this document, such as debugging and language expressiveness. [nikomatsakis], from Amazon Web Services, and [tmandry], from Google, are driving the [async fn in traits initiative](https://github.com/rust-lang/async-fundamentals-initiative/issues/5), which will unlock the ability to write `async` methods of traits. [mw], from Microsoft, is driving the [async crashdump initiative](https://rust-lang.github.io/async-crashdump-debugging-initiative/), which will enable developers to understand the control-flow stacks encoded in crashdumps for their async Rust programs. There is a ton of other work being done in the Async Rust space. Check out the [Async Vision web site](https://rust-lang.github.io/wg-async/welcome.html) for more information. ### Debugging Initiatives (👩‍💻) [wesleywiser], from Microsoft, and [pnkfelix] are spinning up a wg-debugging working group. It will cover at least the following sub-items: improving Rust's debuginfo quality ([mw], [wesleywiser]), supporting split debuginfo ([davidtwco], from Huawei R&D UK), and better integration with trace-based debuggers like `rr` ([pnkfelix]). Immediate goal for this initiative: establish the working group, determine priorities for the backlog of debugging issues, and find out what active users of debuggers miss most when they operate on Rust code. ### Compilation Time Initiatives (👩‍💻, 🛠️) The Rust compiler's end to end latency is known to be a problem. [lqd], sponsored by the Internet Security Research Group, is dedicating the majority of 2022 to working on this, partnering with Rust's compiler-performance working group as well as performance experts like [nnethercote] (from Futurewei Technologies). [lqd] has their own [living document](https://hackmd.io/3Dp68rTDSpWvRDfWF6lbMw?view) that lists areas under investigation. [ISRG]: https://www.abetterinternet.org/ ### Expressiveness Initiatives (👩‍💻, 🦀) A common refrain we hear is: "I need feature X, but it's not implemented in rustc or stable." Many of those features have dedicated owners. For example, Generic Associated Types, or [GATs](https://github.com/rust-lang/generic-associated-types-initiative/issues/4), are an ongoing effort owned by [jackh726]. GATs have many applications, such as traits whose associated types have lifetimes tied to the local borrowing of the receiver type ([e.g. `StreamingIterator`][GAT-motivation]. [GAT-motivation]: https://github.com/rust-lang/rfcs/blob/master/text/1598-generic_associated_types.md#motivation [`async fn` in traits](https://github.com/rust-lang/async-fundamentals-initiative/issues/5) is an ongoing effort (already mentioned above) owned by [tmandry]. This is one of the most frequently requested features for async rust: supplying first class support for traits like `trait Foo { async fn bar(&self); }` The [safe transmute](https://github.com/rust-lang/lang-team/issues/21) project, led by [jswrenn], is expected to be feature-complete in summer 2022. It will enable a large class of types to be transmuted (i.e. zero-cost type conversion) without any risk of injecting undefined behavior. ### Librarification Initiatives (🛠️) These are initiatives dedicated to the "librarification" of the compiler: breaking the monolithic code base of `rustc` into a set of decoupled parts that can be independently developed, and, ideally, repurposed for other kinds of tools besides `rustc` such as `rust-analyzer`. #### Chalk [Chalk] is a reimplementation of Rust's trait system using declarative logic rules, a la Prolog. [Chalk]: https://github.com/rust-lang/chalk Chalk has been years in development, and has been experimentally integrated into rustc in the past. This year, [jackh726] and [nikomatsakis] are owning the task of improving the chalk integration, to drive it to the point where the team can consider migrating to chalk as the implementation of the trait system. This will unlock many features that up until now have been too difficult to implement in the old trait system implementation, and its declaration structure will provide a proper foundation for people to reason about the *correctness* of the trait system. If you want to help out with this, reach out to [jackh726] and [nikomatsakis]. ## Aspirations We would love to get help in any of the areas listed on this document, but this section specifically lists areas where we know we lack resources today. If you are interested in helping with any items here, please do reach out to the owner listed; they will be thrilled to talk to you. ### P-high Aspirations (🦀) [pnkfelix] and [wesleywiser], as team leads, are deploying processes to help us get a handle on the "high priority, but *not critical*" issues that the compiler has accumulated. We will be gradually identifying owners for each who will move progress forward, and in general working to keep better track of the set overall. If you would like to help with the task of reviewing or resolving such issues, reach out to [wesleywiser] and [apiraino], who are co-leads of WG-prioritization. ### Debugging Aspirations (👩‍💻) We want better integration, at least with the popular debuggers. We want to add e.g. expression evaluation support. We also want to reevaluate our debugger extensions for data structures rendering. We have direct contacts with experts in Windows-based debuggers and in LLDB debugging. We do not have any local expertise with extending `gdb` itself, nor any direct contacts with such expertise.  If you want to help out here, please reach out to [pnkfelix] and [wesleywiser]. ### Faster Compilation Aspirations (👩‍💻, 🛠️) #### Parallel Compilation Parallel Compilation is one avenue for improving compiler performance. It is also a very complex area, especially when it comes to the tradeoff of how much of a hit one is willing to take on single core machines with in order to enable more parallel computation. This is an area we think needs long-term collaborative effort with the compiler team. We do not expect to deliver a solution here this year. If you want to discuss more with us about past attempts and ideas for the future, please reach out to [pnkfelix] and [wesleywiser]. #### Incremental Compilation Aspirations Incremental compilation performance and stability are both ongoing concerns to the team. We *know* there is significant room to improve the effectiveness of incremental compilation, in terms of reducing the amount of redundant work done by successive `rustc` invocations. In addition, there is a significant amount of work that could be done to improve our testing infrastructure for incremental compiliation which does not require deep knowledge of the compiler. If you want to learn more, reach out to [cjgillot] and [Aaron Hill]. #### Inter-crate Sharing Aspirations nnethercote has noted that there may be opportunities to improve end-to-end compilation time for multi-crate builds by identifying redundant activity that can be shared between builds of distinct crates. (For example, the metadata from libstd is read and decoded on every single crate compile.) If you are interested in exploring this idea further, reach out to [nnethercote] and [lqd]. ### Expressiveness Aspirations (🦀, 👩‍💻) const generics and const eval are making steady progress. There are a *lot* of feature flags, which implies there's a lot of knobs that could be turned on and off. What we can probably use the most help with is in identifying what subset of the features we should be striving to stabilize in order to unlock specific use cases for Rust developers. So, if you or your team is enthuastically awaiting const generics or const eval, reach out to [lcnr] (supported via [sponsorship][sponsor-lcnr]) and [oli-obk]. [sponsor-lcnr]: https://lcnr.de/funding/ ### Librarification Aspirations (🛠️) #### MIR tooling Various stakeholders, especially in the formal methods space, are making extensions to Rust that are based on analyzing MIR, the intermediate representation used by the compiler. Should we be trying to stabilize that as an interop format of some kind? Reach out to [pnkfelix] and [wesleywiser] if you are interested in helping us here. ### Compiler Backend Aspirations (🛠️, 👩‍💻) #### Ease writing new backends One source of tedium when defining a new Rust compiler backend is implementing the intrinsics that each backend must provide. But a small change to the intrinsic system: namely, allowing intrinsics to define a [fallback MIR implementation][], could ease that burden. Reach out to [scottmcm] if you are interested in helping out here. [fallback MIR implementation]: https://github.com/rust-lang/rust/issues/93145 #### Cranelift The [Cranelift Code Generator][Cranelift] is getting a lot of attention from various parties. rustc has a [Cranelift backend][]. If you are interested in helping out with it, reach out to [bjorn3] (supported via [sponsorship][sponsor-bjorn3]). [sponsor-bjorn3]: https://liberapay.com/bjorn3 [Cranelift]: https://github.com/bytecodealliance/wasmtime/tree/main/cranelift [Cranelift backend]: https://github.com/bjorn3/rustc_codegen_cranelift ### Diagnostics Aspirations (👩‍💻) The Rust compiler has pretty good diagnotics. But the good news is, there's a [full employment theorem](https://en.wikipedia.org/wiki/Full_employment_theorem) for diagnostics engineers which is supported by the 1,500+ [open diagnostics issues](https://github.com/rust-lang/rust/issues?q=is%3Aopen+is%3Aissue+label%3AA-diagnostics) we have. Diagnostics improvements are an *excellent* first step for learning about how to contribute to the Rust compiler. If you're interested in helping out but don't have any idea where to start, fixing diagnostic bugs is a great jumping off point, and you can reach out to [estebank], at Amazon Web Services, to find out more about how to help. ## Conclusion Reading over this list, the number of items on it seems quite daunting. Do we really think we can get all this stuff done in one year? No, we don't! :laughing: The introduction explicitly said the latter half are things that *don't* have resources attached to them. And the word "aspiration" was chosen to reinforce that. This is where you all, the Rust community (including *future members* of that community) come into the picture. Each item has one or two people listed with it; if you're feeling inspired, please do contact us!