---
title: T-spec meeting 2024-03-21
tags: ["T-spec", "meeting", "minutes"]
date: 2024-03-21
discussion: https://rust-lang.zulipchat.com/#narrow/stream/399173-t-spec/topic/Meeting.202024-03-21
url: https://hackmd.io/jR6-og7YQ9WFIz1wpaDqhw
---

Attendees: Joel Marcey, Mara Bos, TC, Lukas Wirth, Eric Huss, Connor Horman, Pietro, Urgau, pnfelix, Mario Carneiro, Monadic Cat

Potential Agenda:

* Test Suite and Code Samples
* Any other business

## Test Suite and Code Samples

https://github.com/rust-lang/spec/issues/49

Joel: (summarized the issue created by Eric)

Eric: Is this interesting to the specification effort? If so, we can talk about how this works? (e.g., Ferrocene - links to the rustc test suite). Prefer to have test suites more local - inline.

pnfelix: Concerned that creating a test suite would be a lot of effort. Would prefer to use tests from rustc or add to rustc. Strongly advise against having a separate test suite. And strongly advise against not having the project manage it. Possible downside is that if the project manages it, they may modify things that we do not want.

Pietro: It was less effort to link to the rustc test suite for the Ferrocene specification. What do we want from this test suite upstream? Agree with pnfelix that we don't create our own suite.

Joel: How amenable is the compiler team to adding new tests?

pnfelix: Generally amenable. Considerations for tests that may be expensive to rust. Downside is that time it takes to get new tests PRs merge by the compiler team.

Connor: Very interested in the spec having a test suite. It is not a high priority that we have a comprehensive test suite that is specific to the specification. Consider having our own test suite in the future.

Joel: Believes momentum of this decision and priority of other items will make it difficult to reverse course.

TC: Personally, I'm sympathetic to the concerns about building a new test suite. But I want to be sure that Eric is heard on this. He raises a number of considerations we may want to discuss here. E.g., it sounds like Eric is proposing something similar to doc tests. He's suggesting that these be tightly integrated and shown in the rendering of the spec itself. He points out that these tests, in his experience, have to be written when writing the spec anyway for two reasons: 1) for the writer, to verify understanding, and 2) for exposition to the reader. He highlights that not putting these tests in the spec itself puts the spec in an awkward position of having to chase down changes to these tests.

Eric: In Mara's sample chapter, there was sample code inline. And it is important that we have that in the specification. Personal experience is that it is difficult to find exisitng tests that are testing what is being written (say for the reference).

Pietro: Limitation of linking to the tests in rustc. Only usable with the rustc that that the tests came with. Diagnostics can change between versions.B elieves that separating out the existing test suite into its own spec test suite will not be so hard because we will already have the tests. For spec items that were not covered, kept a log file.

pnfelix: Allocate a spec subspace in the rustc test suite. Still part of the CI for the compiler. Can simplify that we have only care about error code and not diagnostics.

Joel: Duplication of tests with a subspace?

pnkfelix: Shouldn't be an issue.

Connor: Integrated tests directly in the spec. We are going to lead a lot more capability to do that effectively. Layout tests, for example.

TC: I'm curious to what degree we may perhaps be responding to the specific words "test suite". If Eric had instead proposed certain "doc tests" within the spec, would we still have these concerns and want to push all of these doc tests into `rustc`? Are there any doc tests that we want within the spec itself?

Pietro: Amount of work is the same whether tests are inline or separate.

Joel: Being using the existing test suite, but be congnizant at some point that we may need to go down our own route.

TC: The premise of Eric's proposal seems to be that we need to write many tests anyway, 1) for our own understanding, and 2) as necessary expositional material. Everything else in his proposal seems to fall out from that. I'm curious whether we disagree with the premise or with the proposed consequences of it?

Connor: Agree with the premise.

pnfelix: Don't disagree with the premise. The main problem with linking is that there is not easy ways to show certain benefits of the tests within the specification.

Eric: Don't agree that it would be that much work. But if others feel that way, we could try linking. Trying to test with rustdoc is not very satisfying.

Connor: Problem with inline tests would be tooling

pnfelix: Does Eric believe that the current rustc test suite does not have the right level of validation?

Eric: Issues with tests as written. With current tooling to test Mara's code samples, all we have is rustdoc. In the reference, a lot of tests are ignored or not validating what they are showing.

Pietro: We are not on the same page of what this test suite is for. The question is less make sure that the samples in the spec are valid. Is the goal every part of the specification is testable.

Eric: Won't have 100% coverage. But we will have illustrative examples. And a second kind of test - making sure we are covering what we say in English are actually true with tests. These would be linked.

Mario: Development differences between using the tests in rustc vs. having them in the spec. Scheduling differences for example. rustc = nightly. spec = specifying older versions. How are we going to handle if we are running UI tests in the repo?

Eric: Spec will follow the same process as the reference. Following the current master branch on what rust is.

Mario: Does that mean a previous version won't be specified?

Eric: Every rust spec will have a version. But we will start at a specific version.

Joel: Don't we want the specification to be the source of truth for rustc and thus its test suite should be the source of truth?

Pietro: Yes, but you can still share test suites.

pnfelix: Coverage and exhaustiveness should be in the main test suite. Illustrative examples would be hard to live in the rustc test suite. Having a separate test suite for that case makes some amount of sense. Worried about divergence.

Mario: Normative tests is a good point. Linking to tests could be problematic because how would folks know what the "golden test".

Pietro: Links are defined in the tests themselves.

Eric: We should move forward to linking to the rustc test suite in some way. But a little uncertain on how we handle illustrative examples.

Pietro: illustrative examples live in the spec repo.

---

## Chat Content

https://hackmd.io/jR6-og7YQ9WFIz1wpaDqhw

8:01
avatar
Connor Horman
Connor Horman says:(FTR: I am working on a blog post that is pointing out that we should take an approach to Alternative Implementations that doesn't exclude us from supporting them in the future)
8:22
MC
Monadic Cat
Monadic Cat says:As a bystander, I think it makes sense to get spec tests their own space in the rustc test suite, as the shortest path to having rustc ensure it passes all of them on every release. But I also think there's room for just having a tool for running every sample Rust snippet presented in the spec, to make sure spec writers don't woopsie.
8:30
avatar
Connor Horman
Connor Horman says:One place I'd like to use internal tests is the layout chapter, to validate the places I use a struct layout for a type.
8:30
MC
Monadic Cat
Monadic Cat says:In my opinion, writing a tool that extracts every Rust snippet from a collection of markdown files shouldn't be that hard... having used pulldown_cmark exactly once, lol
8:34
avatar
Connor Horman
Connor Horman says:Having used it twice, it would not at all be hard. The fancier tooling for more interesting tests would be harder.
Connor Horman says:(Running a comprehensive test suite might be a custom mdbook backend)
pnkfelix
pnkfelix says:Validation of Illustrative examples, versus Coverage of every concept
8:38
U
Urgau
Urgau says:Couldn't the spec tests be subtree of the rustc test suite ? That way it would integrate seemlesly.
pnkfelix
pnkfelix says:Urgau: That is what I have been trying to suggest. I guess I have not been clear.
being able to run a test suite against multiple distinct versions of rustc is perhaps the best argument I see for making a test suite that is decoupled from the rustc test suite.
pnkfelix says:(but I think that can be accomplished via swapping a subtree within the rustc ui test)
Connor Horman
Connor Horman says:> being able to run a test suite against multiple distinct versions of rustc is perhaps the best argument I see for making a test suite that is decoupled from the rustc test suite.

Or against !rustc. This is why I want it in the future, but not necessarily as high priority now.
On the side of desyncing spec->rustc vs. rustc->spec, rustc could submodule the spec repo or a separate spec-test repo.
8:55
P
pnkfelix
pnkfelix says:We have a toml files that can assign ownership of the subtree
to ensure that a member of T-spec has to sign off on changes to files that live underneath the subtree
pnkfelix says:(or at least are notified about them)