# W3C Solid Community Group: Test Suite Panel

* Date: 2022-06-01T11:00:00Z
* Call: https://meet.jit.si/solid-test-suite
* Chat: https://gitter.im/solid/test-suite
* Repository: https://github.com/solid/test-suite-panel
* Status: Draft

## Present
* [Sarven Capadisli](https://csarven.ca/#i)
* Pete Edwards

---

## Announcements

### Meeting Recordings and Transcripts
* No audio or video recording, or automated transcripts without consent. Meetings are transcribed and made public. If consent is withheld by anyone, recording/retention must not occur.
* Join queue to talk.

### Participation and Code of Conduct
* [Join the W3C Solid Community Group](https://www.w3.org/community/solid/join), [W3C Account Request](http://www.w3.org/accounts/request), [W3C Community Contributor License Agreement](https://www.w3.org/community/about/agreements/cla/)
* [Solid Code of Conduct](https://github.com/solid/process/blob/main/code-of-conduct.md), [Positive Work Environment at W3C: Code of Ethics and Professional Conduct](https://www.w3.org/Consortium/cepc/)
* Operating principle for effective participation is to allow access across disabilities, across country borders, and across time. Feedback on tooling and meeting timing is welcome.
* If this is your first time, welcome! please introduce yourself.

### Scribes
* Sarven

### Introductions

---

## Topics

### About

* SC: The Test Suite Panel is a "horizontal" panel where test software developers, spec contributors, implementers, and any other group can join to discuss/collaborate.
* SC: This meeting is intended to focus on the development of a Charter for this panel. Continues from [2022-05-13 meeting]( https://github.com/solid/test-suite-panel/blob/main/meetings/2022-05-13.md) where we focus on [whether there is accurate and sufficient information for the Charter, and no major disagreements](https://github.com/solid/test-suite-panel/blob/main/meetings/2022-05-13.md#proposal-4-review-results-of-proposals-1-to-proposals-3).
* SC: We may need to revisit due dates for PROPOSAL-5 and PROPOSAL-6 since we had to postpone the meeting for review/approval - due today - for a few days due to holidays and scheduling with participants. We are still fairly on track though.

### Specify requirements and criteria for publishing test summaries and reports
URL: https://github.com/solid/test-suite-panel/issues/5

* SC: Everyone reviewed? Let's start with any major concerns/objections, if any.
* PE: Finally got there but one comment from MJ that we didn't get to.
* MJ: Happy with the way we publish with current text - notify them, have a chance to object, if objections, they can provide text and that will be included in the report. It leaves it open, what if they say, I dont want you to publish with a comment "don't publish at all".. what do we do then? I think it is useful to refer to priority-of-constituencies.
* SC: Could you clarify that order that you were thinking of for the roles/stakeholders (most important to least..)
* SC: Principles originally derived from https://www.w3.org/TR/design-principles/#priority-of-constituencies
* MJ: From solid/process:
>User needs come before the needs of web page authors, which come before than the needs of user agent implementors, which come before than the needs of specification writers, which come before theoretical purity.
* SC: So I think that was in context of spec development when I originally recommended to be in the process. Can we approach this from the point of test-suite work? I'd assume it'll be different.
* MJ: End-users/Organisations -> App developer -> Server implementer -> Spec editors -> Theoretical
* PE: End users are theoretical priority, but in reality won't we resolve issues at a lower level before end users are impacted?
* MJ: There's been cases in the past..
* SC: I was thinking along these lines for test suite work/reporting:
* SC: Spec Implementer -> Spec author -> Test review/author
* SC: e.g., who is the report for? consumer of the report. I'd assume it needs to help the software/project maintainer (spec implementer).
* MJ: We shouldn't censor the part where it helps the broader community about the findings in the report.
* MJ: We want our best information to be live but notify everyone to give a heads up and let them give a remark e.g., implementer can say "we misinterpreted the spec but it'll be there next time". That's how I forsee it going. There'll be situations where they say "we are failing that test due to security bug.. please don't list our data".. and this is where the other interests are more important re vulnerability. Once everyone has a chance - the implementer updates with a patch... once the fix is available and public, then we can update the report.
* MJ: What we want to avoid is people blocking.
* MJ: if they don't want to publish, they can write it down - be specific.
* PE: That sounds reasonable just need to capture the process - the one decision making when we'll withold reports from being published.
* SC: How much of that should go into the charter and also to be worked out depending on case?
* PE: Is it sufficient for the charter to say - the processes are defined in the panel, including the decision policy. The charter can capture the intent. The purpose of this panel is to write tests and produce reports.
* SC: I'm okay with that. That is also one of the subjects of issue 7. So we do have some stuff that needs more detailed process documented. Especially the checklist in 7, needs to be extracted into its own document where test reviewers go through when they're looking at a proposed test.
* SC: Can we have separate ACTIONs to extract process related stuff from issues 5-7 into separate documents?

ACTION: Create process documents. Panel (maybe SC) / TBD.

* SC: Anything else in issue 5?
* MJ: Not give expectation for server implementers that we are not going to publish unless they agree. Default is we publish everything but include statements. In specific cases we may withhold some test results based on requests from server implementers.
* SC: Sure, to be worked on in separate process document.
* SC: Listen to implementor but.. I preassume if tests are approved, there shouldn't be major concerns about publishing the report. Let's revisit.
* MJ: Identify the reasonable/legitimate reasons.

### Specify requirements for test suite and test metadata
URL: https://github.com/solid/test-suite-panel/issues/6

* SC: Everyone reviewed? Let's start with any major concerns/objections, if any.
* PE: They're reasonable expectations.
* MJ: No objections.

### Specify criteria for assessing tests and results
URL: https://github.com/solid/test-suite-panel/issues/7

* SC: Everyone reviewed? Let's start with any major concerns/objections, if any.
* PE: I was happy with this list.
* SC: Again, this will move into its own document for reviewers.
* PE: Basically happy, some TBDs but happy.
* MJ: Looks good.

### Proposing the Charter
* SC: Need to revisit date. Supersedes https://github.com/solid/test-suite-panel/blob/main/meetings/2022-05-13.md#proposal-4-review-results-of-proposals-1-to-proposals-3
* MJ: Pick a date.
* SC: 2022-06-07?
* SC: No objection to me as continuing as chair?
* MJ: Happy that you're chairing.
* PE: Okay.

ACTION: SC

### Aside
* MJ: Happy with one panel and one charter. Good chat with Pete on removing "independent". Won't publish specification-tests directly so doesn't look like we're publishing prematurely. Now we publish based on Jest tests and work towards entire spec covered and those tests approved. One day have approved tests and later return jest tests. Hopefully they'l lbe published on solidproject.org and retire solidservers.org - checked and quality assessed. We have a good way forward. Had a session with M before this. Y was also contributing. Which we can hopefully retire ..