---
title: T-spec meeting 2024-10-31 🎃
tags: ["T-spec", "meeting", "minutes"]
date: 2024-10-31
discussion: https://rust-lang.zulipchat.com/#narrow/stream/399173-t-spec/topic/Meeting.202024-10-31
url: https://hackmd.io/UqCAyaZuRIqMfBmUpoGQKA
---

🎃 👻 Halloween Edition

Attendees: Joel Marcey, Connor Horman, Mara Bos, Moandic Cat, TC, Sid A, Niko Matsakis, (pnk)Felix

Regrets:

Agenda:

* Conformance Testing
* Lexing
* If Time: Spec output formats

## Conformance Testing

https://rust-lang.zulipchat.com/#narrow/channel/399173-t-spec/topic/Conformance.20Tests

Connor: Annotate the UI test suite with different reference test. Found the process to be more of a scavenger hunt. Hard to find which tests line up since the suite is disorganized.

Eric Huss recently posted: https://github.com/rust-lang/rust/pull/132376 (see: https://rust-lang.zulipchat.com/#narrow/channel/399173-t-spec/topic/test.20annotations)

<https://github.com/lccc-project/rust-spec-conformance> lccc project started on this.

TC: Part of the plan is to reuse the FLS annotations. Those don't point to specific paragraphs but only different chapters, but it's a place to start, and Eric seems to have a clear picture of how to go about this.

Connor: Not sure if the FLS annotations will make a substantial difference, although they haven't fully explored them.

Niko: Good to ask wehther we can port over the FLS approach, but will it be valuable. Can't really tell what I am supposed to take from a test. Shows basic shape. But does it cover an edge case?

TC: Eric has proposed in the past doing our own test suite. But we had discussed this, and we had decided to use the UI tests. Eric has since built tooling in this direction, and we've begun to merge annotations into the UI test suite.

TC: This work may focus the mind on making the UI test suite better. If we did our own, how different in purpose would that be than the UI test suite?

Connor: UI Test suite has more than conformance and has things unrelated to the specification.

pnkfelix: Having own test suite gives freedom to deliver tests to what the spec needs. Another benefit is that we don't have to deal with the GitHub process landing tests in the UI test suite.

Connor: Hadn't considered the GitHub process. Own test suite can be reviewed by `t-spec`.

Connor: An advantage I have in mind is that a separate test suite could be used outside of the Rust project by other implementors of Rust, and that this would give them an incentive to contribute to that suite.

Mara: It is important to realize how important the UI tests are from what we need for the specification. They are more about diagnostics than guarantees. For the spec we are looking for more guarantees. Two completely different goals.

Niko: Useful to focus the conversation on what we need and how it overlaps with the existing UI test suite. Articulate what we need with the test suite, how that might be in conflict with the current test suite, and go from there.

pnkfelix: Is the right thing a separate repo that is not sync'd with the compiler UI test suite? Or should we have our own that is sync'd with the UI test suite, like a mirror?

Connor: Plan is separate repo that gets submoduled into other test suites.

TC: Why do you (Connor) want it to work that way? What purposed is not served by creating a subdirectory in the test suite and putting the conformance tests there?

Connor: To support alternate implementations and throw submodules in other projects. Also the review process.

Mara: We could be working on a shared worktree. Git subtree is a good tool here, with our own place and review process. And create a PR to integrate into the rust-lang respository.

Mara: I think the ideal future would have have separate ui and conformance tests. If we do that properly, then we can have a basic rule for changes to rustc: if it changes the conformance tests, it needs a lang FCP; if it only changes ui tests, it doesn't.

Niko: Mara identified some advantages.

Niko: Maintain the spec on a rolling basis. A change to the compiler would have a "spec buddy" to help write tests with edge cases. Spec author can guarantee good test coverage.

TC: I'm unmotivated by the idea of trying to accomdate other implemenations here. It's difficult enough to do things in support of our own interests. We're trying to get to a certain place as quickly as possible, in terms of having test coverage to close the real and perceived gap of use cases that the Reference does not cover. I'm cautious about adding complexity that might increase the distance to that goal.

Connor: Differing views on what the shortest path is. The work I am doing is easier to get more tests annotated. Easier to keep text and tests in sync if tests in their own repo.

pnkfelix: There is a difference between putting a PR on a separate repo to a bot flagging a potential change.

pnkfelix: The cleanest simplest thing is the for Project members to utilize something they are familiar with, which is the current test suite.

Mara: Prefernce to put this in the Rust repository in a separate directory. We can easily get r+ permissions for the spec team. Merging a PR in the rust repository is not hard.

Connor: Experience that it is harder to manage PRs in the rust repository.

Mara: Not my experience at all.

pnkfelix: What about a world where the root source of truth in rust-lang/rust and then mirror to another repo. We can then port PRs at the mirror to rust-lang/rust.

Connor: Believes the UI test suite is disogranized and don't want to add to the disorganization.

Niko: Back up, go over the notes, figure out the goals and come up with a revised proposal. Writing new tests might be faster than mining.

Joel: I think we are all converging, maybe save for Connor, that we should utilize the existing UI test suite infrastrucutre in some capacity. Probably a subdirectory. But to help compromise with Connor, we can maybe use the mirror idea.

Connor: Willing to agree with that, but want us to consider the future here, and not set us up for redoing this work in 2 years.

Mara: Maybe not a subdirectory in UI, but a same level directory instead. Small change. `tests/ui` and `tests/conformance`. Can use different tooling/organisation.

TC: I'd prefer to use a subdirectory because that is what the docs say and is what people run commonly. I want the spec tests to be run as part of this normal testing.

Mara: Put conformance tests in separate dir outside UI tests, but running UI tests also runs conformance tests and puts stderr in ui/ directory. So stderr of conformance tests is part of the ui suite. Conformance suite only does pass/fail checking (and maybe some error code).

Niko: Separate out what we think our the conformance tests without reinventing the wheel.

TC: Concrete proposal for moving forward: when writing tests, make PRs to the UI test suite and make annotations to connect them to the reference. For new tests, it's OK to use a new subdirectory under `tests/ui/<newdir>`, and that's where they'll live until and unless we develop tooling that might justify them living in `tests/<newdir>` instead (we can always do a `git mv` at that point).

TC: When you write these PRs, please CC the spec team. Most here have the ability and context needed to `r+` these.

## Lexing

https://rust-lang.zulipchat.com/#narrow/channel/399173-t-spec/topic/Lexing

## Spec output formats

## Meeting Chat

Monadic Cat
Monadic Cat says:hellooo
8:01

Monadic Cat says:does that mean there's time for me to go grab a sandwich without missing anything
8:02
Monadic Cat says:lol
8:02
Monadic Cat says:amazing, brb
8:02
avatar
Niko Matsakis
Niko Matsakis says:I was waiting in a google hangout for a while
8:05
Niko Matsakis says:😃
8:05
MC
Monadic Cat
Monadic Cat says:i am back, only slightly late
8:07
Monadic Cat says:lol
8:07
avatar
Connor Horman
Connor Horman says:Feix, Mara, Niko was the order.
8:13
me says:I assume if we had our own subdirectory in the UI test suite, we would still have to follow the current GitHub process for the entire test suite?
8:16
avatar
Connor Horman
Connor Horman says:The Minimum Required Diagnostic is "Segmentation Faul (Core Dumped)"
8:18
avatar
Mara Bos
Mara Bos says:👍
8:18
me says:if we had our own test suite, would that add additional process for those outside the spec wanting it to use it for different purposes. e.g., compiler change requires both an update to the UI test suite and this new specification test suite?
8:20
avatar
Connor Horman
Connor Horman says:I'd expect new language features need new conformance tests, but a compiler change that isn't a language feature wouldn't.
8:21
avatar
Mara Bos
Mara Bos says:👍
8:22
avatar
Niko Matsakis
Niko Matsakis says:pnkfelix: you are saying "mirror", right? not "MIR"?
8:22

me says:I kinda like the mirror idea.
8:22
MC
Monadic Cat
Monadic Cat says:git subtree can sync a directory in both directions, iirc, so you should be able to make it live in rust-lang/rust and still have people work on it from outside? but I'd have to review to say how easy it'd be
8:26
PF
(pnk)Felix
(pnk)Felix says:we can give t-spec members review rights for a relevant subdirectory
👍
8:26
(pnk)Felix says:our review system has support for that
8:27
me says:Right pnkfelix, that goes to my question above.
8:27
me says:If we could get a subdirectory with our own review process, that makes this a lot easier to digest.
8:27
me says:I think we are converging that we at least need a separate place for conformance tests. The question is whether it lives with the current test suite in a separate subdirectory or in its own repo that can be easily submoduled.
8:32
avatar
Niko Matsakis
Niko Matsakis says:(is somebody keeping notes on this and, if so, where?)
8:33
avatar
Connor Horman
Connor Horman says:
https://hackmd.io/@rust-spec-team/HyMN-bgb1l

8:33
me says:
https://hackmd.io/UqCAyaZuRIqMfBmUpoGQKA?edit

8:33
avatar
Niko Matsakis
Niko Matsakis says:thanks! I was htinking we've uncovered a lot of interesting constraints and Connor I'd be up to work with you on thinking out their implications
8:34
MC
Monadic Cat
Monadic Cat says:I think the separate directory is a good idea, the automatic "this is a language change" classification for PRs that have to hit it sounds nice, the official subtree-ing does seem like jumping the gun
8:35
me says:We would just need to make sure we aren't bogged down by a whole lot of process if we want to get a test added.
8:37
me says:Remember separate implementations is not yet a goal of our work, via the RFC.
Mara Bos
Mara Bos says:👍
8:40
T
TC
TC says:👍
8:41
MC
Monadic Cat
Monadic Cat says:inb4 you end up merging the reference into the rust-lang/rust repo (this is a joke)
8:41
T
TC
TC says:The style guide is merged; it's not beyond consideration.
8:41
MC
Monadic Cat
Monadic Cat says:owo
8:42
me says:Our current spec scope is to support rust-lang so it behooves us to do work to support them as much as we can (vs. other implementations)
Connor Horman
Connor Horman says:That's why my point comes with the offer of external resources from that.
8:43
PF
(pnk)Felix
(pnk)Felix says:Ah the old "this should be an RFC but there's too much uninformed eyeball traiffc on rust-lang/rfcs repo" problem
8:45
avatar
Connor Horman
Connor Horman says:Not so much uninformed, but not necessarily individually helpful.
8:45
MC
Monadic Cat
Monadic Cat says:"root" works, so does "main"
8:45
me says:👍
8:46

me says:I agree with the subdirectory approach. And pnkfelix's idea of a mirror. I just want to see if we can define our own review process for the subdirectory if we find that the current review process is to onerous.

Monadic Cat
Monadic Cat says:woops
8:50
PF
(pnk)Felix
(pnk)Felix says:I don't mind taking a path that makes it easy to "undo" the choice here. My gut is that the subdirectory approach will be inherently easier to undo than starting with the separate repo approach.
(*either* variant of the subdirectory approach, be it `tests/ui/subdir` or `tests/subdir`)
(pnk)Felix
(pnk)Felix says:we can update the dev docs
8:55
(pnk)Felix says:in many ways, running the conformance tests *first*, before ui, is teh **best world outcome*
8:55
avatar
Connor Horman
Connor Horman says:^
8:55
(pnk)Felix
(pnk)Felix says:i.e. find out about all the failures to conform before you dig into diagnostic issues
8:55
avatar
Connor Horman
Connor Horman says:I'm not sure I care if I broke a diagnostic somewhere if I broke the language.

Niko Matsakis
Niko Matsakis says:(tests that contain a `SPEC` annotation might also not have `ERROR` lines)
8:59
Niko Matsakis says:those are kind of optional etc
Mara Bos
Mara Bos says:👍
8:59

avatar
Niko Matsakis
Niko Matsakis says:the main thing is that we label the test with what outcome we expect
Mara Bos
Mara Bos says:👍
8:59

avatar
Niko Matsakis
Niko Matsakis says:the main thing is that we label the test with what outcome we expect