Design meeting 2024-10-30: Lang team spec process proposal

--- title: "Design meeting 2024-10-30: Lang team spec process proposal" tags: ["T-lang", "design-meeting", "minutes"] date: 2024-10-30 discussion: https://rust-lang.zulipchat.com/#narrow/channel/410673-t-lang.2Fmeetings/topic/Design.20meeting.202024-10-30 url: https://hackmd.io/LNOPgmDsS8-1MKB9spGBTw --- # Lang team spec process proposal ## Goals * For Rust users: * When a new feature is released, I should be able to go somewhere and read a solid explainer. * When I want to understand a feature, I should be able to go to a specification and read about it. * For spec consumers: * I should be able to re-assess my product across Rust versions with a minimum of churn. * For contributors: * I can drive a feature without having to know everything about Rust or be a unicorn developer. * For compiler team: * I should be able to know what to implement and whether a proposed implementation is correct. * For reference team: * I should be able to get a precise description of how a new language behavior will work so as to be able to update the Reference. * For lang team: * I should be able to review precise descriptions of how proposals for new language behaviors would work so as to be able to decide whether to adopt them. ## TL;DR The TL;DR of my proposal is a few changes: * Update or rewrite RFCs when design changes pre-stabilization * Result: It is a ready-made explainer when stabilization occurs. * Define stabilization process as: * Open a "stabilization issue" as a checklist of work to be done; this happens early. * [x] Overview * [ ] Spec text authorship by spec team * [ ] Spec text review by lang team * [ ] Ping compiler + types team * [ ] Tool polish (rustfmt, etc) * [ ] Stabilization PR (compiler team) ## Diagram ```mermaid flowchart TD Experimentation[ Experimentation Requires experienced contributor and lang-team liaison ] RFC[ RFC Describe and motivate feature from user's point of view ] Implementation[ Implementation Implement the feature as designed; resolve unresolved features ] FeatureComplete[ Feature complete Feature works as expected; post blog post to Inside Rust encouraging experimentation 📝 ] subgraph Stabilization process StabilizationBegins[" Stabilization proposal Open an issue with checklist. Indicates someone is keen to drive. "] AuthorSpecification[" Specification authoring Done by the spec team "] ReviewSpecification[" Specification review Done by the lang team ✅ "] Polish[" Update associated tools rustfmt, rust-analyzer, etc "] StabilizationPR[" Approve stabilization PR Reviewed by compiler team ✅ "] StabilizationComplete[" Stabilization complete Available on stable; included in release notes with link to RFC "] end RustBookEtc[ Update to Rust book and other 'laggy' items ] Legend[ Legend ✅ -- FCP decision 📝 -- Blog post ] Experimentation --> RFC RFC -- fcp merge ✅ --> Implementation Implementation -- update RFC with any changes ✅ --> RFC Implementation --> FeatureComplete FeatureComplete -- Expected time: 3 months (?) --> StabilizationBegins StabilizationBegins --> AuthorSpecification --> ReviewSpecification StabilizationBegins --> Polish ReviewSpecification --> StabilizationPR Polish --> StabilizationPR StabilizationPR --> StabilizationComplete StabilizationComplete --> RustBookEtc ``` ## FAQ ### Why go back and update RFCs? In practice people use RFCs as the explainer and it should be up to date. It also means that when we post a blog post, we can link to the RFC without embarassing caveats. ### What about really complex changes? If edits to an RFC are complex, it's ok to deprecate the existing one and just rewrite it. An alternative is to create an RFC around the changes and (in same PR) edit the original text. ### Why is the spec team authoring spec text? It turns out that this requires context. The Reference maintains a particular voice, style, and set of standards, and it can often take more back and forth to achieve that than it would take to just write it, assuming that a complete and comprehensive description of the new behavior is available. # Discussion ## Attendance - People: TC, tmandry, nikomatsakis, Josh, scottmcm, pnkfelix, Eric Huss, Connor Horman, Yosh, Xiang, Eric Holk, CE, Nadri, Mara ## Meeting roles - Minutes, driver: TC ## Goals wording > For spec consumers: > > I should be able to re-assess my product across Rust versions with a minimum of churn. tmandry: What is a spec consumer? What does it mean to re-assess my product? Also, are "contributors" in this section contributors to the compiler? NM: What I meant by spec consumer is... mostly I mean people in safety-critical who are using the spec as an input to their product. Once you've based something on the spec, you want to know what has changed, and you'd prefer the diffs be minimized. NM: The relevance here is that, if we wish for the spec to be a useful product to build on in these kind of industries, we have to pay attention to that. PK: In thinking about the customers here, there's real money involved in how this is done. NM: In a way, we as the lang team as not so different from the assessors. ## High level NM: I'm looking for a firm decision out of this meeting. I want to talk about the options that might work. This is mostly a straw proposal. It may be helpful in thinking about how we manage the spec going forward. The question there are tied to this. In a way, it comes down to deciding whom the customers of the spec are. Are we the customers? I think we are. The recent issue on temporary lifetimes has reinforced this to me. ## Stabilization PRs TC: As a process matter, I really like us FCPing stabilization PRs. We started doing this rather than commonly doing FCPing on tracking issues, and that has worked well. There's kind of no substitute for doing our check at the most final point. It helps to make it very clear what we're accepting and reduces the potential for skew or misunderstanding. In theory, of course, reviewing the spec changes should be enough. I can see a world in which that's fine, and that after that it's not our concern. Maybe what I'm saying is that we should be sure we're in that world before we change the process. Because our language is tied to one implementation for which we're responsible, we're situated a bit differently than those responsible for other languages that might come to mind. We're responsible for accidental stabilizations and places where we underspecified but people have relied on the behavior. Those things become "part of the language" in a way that they don't when there is a language / implementation separation. This I think makes the problem a bit harder for us. NM: Yes, this makes sense to me. What I have felt at times is that there's an ambiguity about reviewing the specification and the implementation. I agree with Felix that sometimes reading the tests has been useful. One of the big pieces of value I'm looking to get from a spec is a tying together of the spec and the tests. I think that the spec text should be evolving like the language evolves. Making one big decision at the end may not be ideal; it'd be better to be incremental. That might push me more toward wanting to OK the final PR. tmandry: I don't think I want to introduce another lang team FCP in series, so we should probably pick what we're doing. NM: Thinking about it now, a good tenant is to KISS. It's probably better all things considered to have fewer steps and to have a smaller change from where we are, as that's usually more successful. Josh: There are a couple of things coming up in the subsequent questions that might simplify. ## Revising RFCs TC: Revising RFCs is something that's long seemed like a good idea to me, but every time I've thought about actually doing it with a particular RFC, it's seemed like a bad idea. The trouble is this. Because the purpose of the document is to motivate and convince, it's often big; it's much bigger than it would need to be to explain something already decided. And all that text is leaning heavily on itself. So when you change things, often a lot of stuff, particularly about weighing various subtle pros and cons from every angle, doesn't make sense anymore, as the balance has changed. But at the same time, it's strange to just delete it, because that stuff was part of the story. Without it, the GitHub thread can stop making sense. (The worst outcome is to update the obvious things and then not update all of the subtle points elsewhere that lean on what was changed, and to thereby end up with a broken and incoherent document. It's particularly easy to do this on a document you wrote long ago, because you've lost your mental index of how and where everything ties together.) We've previously avoided doing new RFCs at the point of stabilization, instead preferring to write RFC-length stabilization reports. We did this for, e.g., RPITIT/AFIT and ATPIT. Maybe (or maybe not) it'd be better to just make RFCs of these (I've thought about doing this, even retroactively, just to give them RFC numbers). The text above talks about rewriting the RFC, and maybe that's equivalent to what I'm saying here, though I'd probably think of it more as just a normal new RFC than as a rewrite. I mostly just want to emphasize that I think revising existing RFCs is probably in general harder than it sounds, and that if we ask people to do it, we'll have to check carefully that these documents were thoroughly updated in all respects -- and that such checking and the resulting back and forth could represent a substantial review load. (When people are updating a document to check a box, there's not the same kind of natural forcing function that there was when they were trying to convince people originally to accept the change.) scottmcm: I wonder if this comes to the different intents of the different parts of the RFC, like we've talked about trying to better separate the "there's a problem to be solved" part from the "and here's the proposed solution" part. It does make intuitive sense to me that part of stabilizing would be updating the Guide and Reference sections, since that's basically what we want for the blog post and specification anyway. But I agree that redoing all the motivation and alternatives parts seems like it ought to be unnecessary, and would really make it harder to see what was actually originally approved in the first place. (It reminds me a bit about how I've been feeling about lint oversight...) Josh: What if our standard practice was not to edit-in-place the *existing* text of the passed RFC, but to instead *append* a section for "changes that occurred between RFC acceptance and stabilization"? Then there's no expectation that everything in the RFC be updated, just that the delta be documented. Connor: I've long wanted this, because if you're citing an RFC to support a feature, that citation could be wrong. But on the other hand, I'm not sure how necessary this is now that Rust is getting a specification, as it could stand in here. NM: I wanted to call attention to the motivation for this change rather than to the details. I think that in practice RFCs are frequently cited. It's not just to explain the behavior, but to explain the motivation and the thinking at the time. It seems that we try to get the best of both worlds, and to try to treat them as both a historical document and an explanation of behavior. What I really want is an "explainer", i.e. a thing that explains to someone who knows how Rust works before, how it works after. I think that's something that we're missing. It feels like that's what an RFC is modulo that it can be out of date. But we could do this in some other way also. Yosh: Something that I've seen in Swift, that I like quite a bit, is that they make sure to backlink from the RFC to the stabilization report. We don't do that. Are there things we could do to make the Reference or the stabilization report more discoverable from the RFC. TC: We link from the RFC to the tracking issue, and from the tracking issue to everything else. scottmcm: I do quite like the erratta we've added to some things, like <https://rust-lang.github.io/rfcs/2115-argument-lifetimes.html>. eholk: In general, I'm more of the view that those are an immutable record of the decision-making of the RFC project. I think of them more as diffs. So I'd like to not see those change too much. I do think that having some addendum section could work. Having something that discusses status, e.g. that this RFC is modified by or superseded by another RFC would be good also. What does the IETF do there? TC: The IETF process is to treat published RFCs as absolutely immutable. They do not change a single byte. They do publish errata, and they link to RFCs that update or supersede an RFC. NM: Having something at the top that tracks the big status items on the RFC could be helpful, as that is often what we point people at. Perhaps when you think that the implementation is done, that would be the time to write up an explainer that could live in the Inside Rust blog post. scottmcm: I'm a fan of adding errata and links to other RFCs. On the blog post, right now the stabilization PRs get a follow-on issue auto-opened for writing up the release notes and the section in the release blog post. Maybe we could integrate something similar here. scottmcm: I agree with what TC's been saying, about how things lean on others in RFCs. It makes me think, if the motivation section no longer applies, maybe it should be a new RFC. Maybe it's more reasonable if we're making small changes to the Reference section. If we end up in a place where the existing motivation doesn't make sense anymore, maybe that should just be a new RFC. Maybe this is what Niko is saying about big diffs. There are phases to a stabilization, and maybe that comes into play here too. NM: A good follow-up on this is to go through some examples of actual edits that we've wanted to make. TC: Another thing to think about here are partial stabilizations. Often, even if what we do really is a strict partial stabilization, it can give it a different flavor from the RFC, and it makes it difficult to point to that document. NM: That's a good callout, and is something we should be sure to write-up somewhere. ## Inside Rust blog post process TC: As a nit, I'd make the Inside Rust blog post outlined above one of the tracked steps in the stabilization process rather than making it a pre-stabilization issue item. That just seems more sustainable as a process matter. NM: +1. ## Blocker for stabilization? Josh: For clarity, is the proposal here to *block* stabilization PRs waiting for specification text to be writen for them? Or is the proposal to keep the spec team in the loop so that they can *start* ensuring specification text exists, but not block on it? I don't think we should treat spec text as a hard blocker; I think we should assume specification work follows closely behind shipping a feature rather than being a blocker for it. TC: Niko's proposal here is to treat it as a hard blocker. That's necessary as proposed here because the way lang would then work during stabilizations is to OK that specification text. Josh: I would propose, here, that we do this in two incremental steps. Let's start with the premise that we want to make sure the spec text is on track to be updated for each new stabilization, but not block on it. Let's see how that goes. If we find, empirically, that it ends up being done at the same time as other requirements for stabilization, we could later decide to start blocking on it. NM: The question I'm not sure about is, if we stabilize something and don't have spec text... the assumption is that reviewing the spec text may find issues, so what does it mean then if we do that later? Josh: If it does catch bugs, then that makes sense. If it's more of a document thing, then that would suggest it should be more of a fast follower. NM: There seem to have been a lot of cases where we did stabilize something and there were some details that we may not have considered. TC: An example I've noticed is that we are not careful about what is handled pre- and post-expansion. When reviewing reference PRs this becomes very apparent. ## When is this all happening? tmandry: When is the spec going to be ready enough that we would realistically consider blocking on updating the spec? Are we starting with a rigorous text or trying to bootstrap our way to one, one feature at a time? TM: I'm wondering about the roll-out process, I'm not clear on when that's expected to happen, or the criteria. NM: I don't know the answer to that. I'm trying to establish more of what we're trying to get to rather than how we get there. Saying, "hey, we want to do this, let's find funding" is more plausible than saying, "if we had funding, then maybe we'd do this." That's part of what's going on in my mind. If we decide that, to do this, we need 2-3 people writing spec content full time, then we can see if we can make that possible, and if it's not, then we can reconsider. Josh: If the concern is that phrasing it in terms of making it incremental would make it less able to be funded, then clearly we should phrase it in the way that's best for getting funding. It's just that we shouldn't block on it until the pieces are in place. tmandry: Is the idea that we'd be starting with a rigorous specification text, or that we'd be incrementally trying to work our way there? NM: That's still probably under discussion. ## Blocking on spec team writing spec text tmandry: Are we not willing to accept contributions for spec text? In every other case we only block on team approvals rather than teams doing work, and I'm concerned about the need for priorities to align. errs: Yeah, I think this is a concern. This seems to put a much higher SPOF on T-spec than any other team in the approval process (T-lang with reviewing and approving a stabilization report, and T-compiler only needing to *review* contributions but not necessarily author them). Connor: I'd assume that the blocker would be the text being written and approved, and it wouldn't physically require T-spec or its direct contributors to author the text. eholk: I'm also concerned about T-spec bandwidth if they are on the critical path of every feature. Perhaps we need some kind of process to train more people to be able to write spec text in the right style and format so T-spec wouldn't need to request so many changes during review. scottmcm: Do we have a committment from the foundation to consistently have someone paid to write ~~reference~~spec text? If we have something like that where anyone can write the first draft with someone expected to have prompt turnarounds for getting the voice right and checking details, that might reduce the concern. NM: I may have stated it too strongly. I don't think that people couldn't write this themselves. It's that we're not expecting or encouraging it. But I do think it requires some amount of funding so that there is some baseline level of support and it doesn't become a massive blocker. TC: The context here is that having multiple people pitch in can be hard. PRs come in that are stylistically wrong and it's hard to push back without that creating net more work than just rewriting it. It's about style, voice, what the content should be, what it should focus on or not focus on, how many are too many examples or too few, what is too explanatory or too "sales-y". Unlike a PR to the compiler, we can't lint, and don't have a type system to enforce things. This puts more load on the reviewers. TC: There's also a standards side to this. This is why e.g. people on the lang team could do it and it'd be fine. It's about having covered all the bases, done one's due diligence, said what needs to said, run by Ralf what needs to be run by Ralf, etc. When something is part of the way to that but not 95%+ of the way there, it eats a ton of reviewer bandwidth in the back and forth. TC: The more we push for the spec to be more rigorous the harder this will get. It's been a challenge to hit the Reference's current standard, and we have been trying to raise it, but the higher we raise it, the more difficult it gets. tmandry: That makes sense. If someone implemented it, aren't they the best to explain what they did? TC: Yes, implementors do need to explain what they did, but not necessarily to write it in the specification. We need it in a doc somewhere -- stabilization report, RFC, wherever -- to write out what it does and to explain it. That's needed for whoever writes the spec text to work from, and it's needed for the reviewers of the spec text to check that it's correct. Eric Holk: As far as standards/style/etc goes, that feels like an opportunity for training. It would be good in setting an example for better onboarding. It's not necessarily the first thing you should do is watch a video on spec training. These should be skills that are teachable. It would be cool if there were a way to say, "hey you are now trained and therefore we are more comfortable with you doing spec text," which hopefully reduces the number of edits that would be required. ## Testing Connor: Would the authoring of specification text necessarily include tests associated with that text, however the spec team decides to handle those? IE. should authoring and tagging (as needed) these tests also be a blocker to stabilization (if the actual text itself is one)? NM: The way that the Reference is moving is to tag tests with individual paragraphs. The FLS has done this at the level of chapters. This is a key thing, I think. Even just checking that we have an adequate number of tests is important. This is something I had in mind in the original stabilization report process. But we've found this difficult. NM: I'd expect the tests to be there by the time that we're going to stabilization. But I think it gets into the other question here, about when the various steps happen. I imagine that spec authors are going to encounter paragraphs and say "where's the test for this" and implementors will be the ones who write it and/or find problems. The more we can integrate spec authorship into the process the better. tmandry: That makes sense to me. I'd put a vote in the column of incremental progress while writing the initial spec. I agree we'd like to get there. If we figure out what we want the spec to say, but we can't find the test currently, maybe we could merge it with a TODO. It'd be good to make incremental progress that adds value to the product without needing to make it perfect in every way. That's my inclination. ## Testing 2 pnkfelix: Where in the process flow above do you expect tests to be introduced+revised+indexed (as associated with a given feature)? In particular, many times when trying to "really understand" a given feature, I find myself first reviewing the tests, and then, if they don't enlighten me, I might then jump to writing my own variants of the tests in a playground. Tests can represent a useful middle ground between the extremes of the RFC-level (and perhaps also spec-level) feature description vs the low-level details of the compiler implementation. pnkfelix: Thinking about it, it doesn't need to be part of the flow of the diagram. As long as it ends up happening, so that by the time the lang team reviews it, there is that set of hyperlinked tests, that sounds fine. ## Testing 3 yosh: Similar to the two other questions about testing, I find myself wondering about validation of the specification. We have a-MIR-formality for type system semantics. And Ralf Jung's minirust for runtime semantics. These formal systems are sometimes also described as "executable specifications". yosh: With our eyes to the horizon, what do we expect the integration between these three (four if we count ferrocene) efforts to look like? I don't expect us to authoritatively answer this now, but I do think there is value in thinking about this. NM: Good question. I don't know the answer. I do still believe in a-mir-formality. I do think it should relate to the spec. The last time I thought about it, my ideal was that the spec didn't lay out teh rules in detail, but did lay out the invariants that the rules needed to uphold, and that a-mir-formality laid out the rules. But now, I'm not sure that makes sense, and we'll need to think more about it. Another way to look at it is as a prototype of the compiler, as an alternate implementation. TC: There's a related question we're considering about how precise a spec should be, even aside from tying it together with things like a-mir-formality. E.g, we have to decide whether it should have formal inference rules like what we worked out for match ergonomics. NM: I would hope that we would include things like the inference rules for match ergonomics. Those are more clear to me than the English version. ## Motivations for design choices NM: Motivation is so key to really understanding the rules that it feels that we need to include that somewhere. However and wherever we put this, we should make sure it's there. It's on my mind. TC: +1. One of the customers of the spec are people trying to evolve the language. And to evolve it, you first have to know why things are the way they are.