# W3C Solid Community Group: Test Suite Panel

* Date: 2022-05-13T14:00:00Z
* Call: https://meet.jit.si/solid-test-suite
* Chat: https://gitter.im/solid/test-suite
* Repository: https://github.com/test-suite-panel
* Status: Draft

## Present
* [Sarven Capadisli](https://csarven.ca/#i)
* Pete Edwards
* Alain Bourgeois
* Yvo Brevoort
* Michiel de Jong
---

## Announcements

### Meeting Recordings and Transcripts
* No audio or video recording, or automated transcripts without consent. Meetings are transcribed and made public. If consent is withheld by anyone, recording/retention must not occur.
* Join queue to talk.

### Participation and Code of Conduct
* [Join the W3C Solid Community Group](https://www.w3.org/community/solid/join), [W3C Account Request](http://www.w3.org/accounts/request), [W3C Community Contributor License Agreement](https://www.w3.org/community/about/agreements/cla/)
* [Solid Code of Conduct](https://github.com/solid/process/blob/main/code-of-conduct.md), [Positive Work Environment at W3C: Code of Ethics and Professional Conduct](https://www.w3.org/Consortium/cepc/)
* Operating principle for effective participation is to allow access across disabilities, across country borders, and across time. Feedback on tooling and meeting timing is welcome.
* If this is your first time, welcome! please introduce yourself.

### Scribes
* Pete Edwards

### Introductions

---

## Topics

* SC: This meeting is intended to focus on the development of a Charter for this panel. Continues from 2022-05-09 meeting.
* SC: The Test Suite Panel is a "horizontal" panel where test software developers, spec contributors, implementers, and any other group can join to discuss/collaborate.

* SC: Brief announcements:

### Minutes of 2022-05-09
URL: https://github.com/solid/test-suite-panel/blob/main/meetings/2022-05-09.md

* SC: The minutes clarify what was noted during the official time of the meeting and afterwards: https://github.com/solid/test-suite-panel/pull/1 (based on revision https://hackmd.io/laVtVsz7QkOmXHrzkWxfdw/revision/1652108116393 ). I think we covered good ground so we don't need to repeat or go into the same level of detail in today's meeting.

### Test Suite Panel Repository README linking to Work Items (pre-Charter)
URL: https://github.com/solid/test-suite-panel/blob/main/README.md#work-items

* SC: Lists current work items. Will update table based on Charter.

#### Panel and Work Item Repository Access Controls
* SC: I'm looking into having something workable for the whole CG. Will keep the panel posted: https://github.com/solid/team/blob/a34d6f7ff50f5b296eacc47c5fe7f6e25ea7f62f/meetings/2022-05-11.md#members-of-the-w3c-solid-community-group-need-appropriate-access-controls-to-repositories
* SC: Role and use of solid-contrib is under review. Temporary location at present - not intended as a location for new repos. Aiming for all CG members to have read access. Editors and authors have higher level as needed.

### Panel's Chat Room Name
* SC: Follows ACTION of https://github.com/solid/test-suite-panel/blob/main/meetings/2022-05-09.md#panels-chat-room-name . Still looking into it. Need to send an email from account (solid org) that created the channel.

### Test Suite Panel Charter
>* SC: We need to stabilise a mutual understanding and a coordination plan. To advance our work, we need to agree on a charter that details its goals, scope, deliverables, decision policy (e.g., along the lines of https://github.com/solid/process/blob/main/notifications-panel-charter.md ).
* SC: In [2022-05-09 meeting](https://github.com/solid/test-suite-panel/blob/main/meetings/2022-05-09.md#panels-chat-room-name) we had rough agreement (but not "ACTION"ed) on SC to create ISSUEs related to PROPOSALs. If everyone is still okay with that, I can do them after the meeting.
* SC: Most of the points below are summaries from prior meeting or common understanding. They're not intended to be exhaustive. We need to be on the same page / rough consensus in order to move on. Do nits later.
* SC: As agreed, exact details of the PROPOSALs are to be worked out in their own issues. Here we focus on committing to *high level* issues/planning that would be useful towards setting the Charter. They will be used as input for the Charter.
* SC: Again, anyone can contribute to the actions (once I create issues and announce).
* SC: We can adjust the date of expected completion.

ACTION: SC to create issues related to PROPOSALs e.g. create an issue, posed in a way people can answer, consensus will be captured into charter.

#### PROPOSAL-1: Specify requirements and criteria for publishing test summaries and reports.
* SC: Publication process (criteria, consent, locations..)
* SC: Editors have recommendations & publishing rules for specs. They might have similar views for reports.
* YB: Is this about this panel working with editors team? A set of rules published by the editors team?
* SC: Yes - looking for consistency. There are guildelines in other areas, we need similar for reports
* MJ: So they will propose requirements and we can say how much is doable
* SC: Yes, we will give feedback on how test software developers can or cannot provide that information. We will engage in a conversation
* YB: This will set out requirements. We will discuss and if we can supply reports that fulfil this, then the reports will be published according tot the rules?
* MJ: And until they will they be published in the current way.
* YB: This is the collaboration between us and solidproject.org?
* SC: Distinguish approved/authoritative and unapproved/to-be-reviewed test reports.

* SC: Clarify things like "How to distinguish between approved and unapproved." and "What's appropriate to publish."
* SC: We will need to make a decision about what is published and where, and with what consent from maintainers.
* MJ: What happens if editors say we need to get consent, yet the panel says no?
* SC: Re process - what would be the reasons to not do that. Is it communication problem or a use case that needs us to do that. Editors will need to consider that and have a balance. It is easier to say whats approve, or has consent. Harder for what is unapproved, work in progress. Is it appropriate to publish that? Perhaps, as long as people can differentiate
* MJ: What if there is a difference of opinion about colour of report? If panel says we will do A and editors team says we must do B - what is the process?
* SC: When we say "panel" it is not just those here but a place for us, editors, implementers to convene. We need to have agreement that everyone can discuss this but at some point the decision needs to be made by those with responsibility for that. Can editors required specific test software - there would need to be a responsible group to decide.
* MJ: We hope for alignment but if we publish something as the test suite panel, then we mean this group. Others outside can disagree but we should feel comfortable with our decision. We have to have agreement as a group
* YB: Can we write down what we dont' currently agree on so we can discuss e.g. format, machine readable, etc. The other question is where to reports end up and what level of consent to we need to such a location. e.g. for solidproject.org we would need external approval but for solidservers.org would we need that? Is there a location for broader, unapproved information? This is one of the major points that we need consensus about.
* SC: Maybe core question is where do we draw the line on our discussion. As far as decision making goes, there may be 50% but what level do we need. We need to identify the authorities for some decisions, taking into account concerns that have been raised. If agreement not reached, do we need to bring in Tim. Normally a chair will identify differences...
* YB: We can identify what we need to discuss
* MJ: Disagree with SC on what we outsource. If we publish as test suite panel - all 6 people should agree. If we can't agree, do we become 2 panels (software & reporting)? PE promised not to publish so we don't want him to feel that his name is associated with something so he breaks his promise. However, MJ would be breaking a promise he made if info was removed to the sponsors of the test suite work. If we can't find a common ground and no overlap of options then we need to be clear who is publishing. No one has to go against what they promised.
* SC: Its not so much outsourcing. Just like spec process on publication rules there are some persistence and quality promises, we need the same type of things for reports. It is not outsourcing. e.g. use of MIT licence - decision is already made. So considerations are made by the editors for all groups. Panel is just somewhere to talk about test suites - it is not an external thing to the CG.
* MJ: It is about which group is perceived as publishing something. If PE syas he is uncomfortable, that is different to someone else saying you shouldn't publish
* SC: Good we are capturing this as we need to answer these questions and find a workable solution for all

* SC: Specify summary/report format and required information in reports.
* SC: Report notifications/request to publish.
* SC: ACTION for Solid Editors Team by 2022-05-27.

* MJ: We will not unpublish
* SC: Until charter we don't have an agreement. Without it nothing is changing
* MJ: Charter will capture change process

ACTION: SC will create issue to resolve these questions and concerns

#### PROPOSAL-2: Specify requirements for test suite and test metadata.
* SC: Need to provide information about the Test Suite Design (metadata, setup, provenance..), how to Run Tests, Write Tests, Administration, Reverting/Corrections, Notify spec contributors, implementers (new/updates to tests), test coverage.
* SC: ACTION for test software maintainers by 2022-05-27.

* SC: This will give an outline but not expecting it to be done immediately. It is a plan to tackle problems
* SC: One area is test suite design and the metadata about the tests, the software, setup, providence, input about environment. Test suite developers will take ownership of providing this information. Write documents about how to run and write tests. What process? How to we notify spec contributors e.g. new test, give feedback, test coverage?
* PE: No concerns about this
* SC: Should all test software have specific characteristics or can there be different categories of test software. This issue will address this.

ACTION: SC create issue for test software maintainers by 2022-05-27.

#### PROPOSAL-3: Specify criteria for assessing tests and results.
* SC: Test Review checklist (all tests, spec references, script tests, visual tests, accessibility..).
* SC: Usually spec authors or editors
* YB: So this touches on collaboration agreement between spec and test panels - the process around that?
* SC: More concretely - we assume the person writing a requirement has a view about how to test it, whether results match expected outcomes. The tests can be written by anyone but we assume the authors either provide this information
* YB: It is like an agreement re what the spec should require in terms of requirements. e.g. every MUST and SHOULD has a link so we can write tests and link to requirements (or 1 to many links). It gives us a way to write a test, say what it covers and therefore what coverage of spec. If I wrote a test for a requirement, what is process to get spec editor to review so it gets used?
* SC: Ref Proposal 2 - test software editors saying how this communication happens
* SC: ACTION for spec contributors by 2022-05-27.

ACTION: SC create issue for this for spec contributors by 2022-05-27

#### PROPOSAL-4: Review results of PROPOSALS-1 to PROPOSALS-3.
* SC: Whether there is accurate and sufficient information for the Charter, and no major disagreements.
* SC: ACTION for Test Suite Panel by 2022-05-30.

ACTION: SC to add to agenda for Test Suite Panel by 2022-05-30

#### PROPOSAL-5: Propose Test Suite Panel Charter.
* SC: Scope and Deliverables won't be ISSUEd. We note here for shared understanding/need. It'll be reflected in the proposal.
* SC: ACTION for SC by 2022-06-01.

ACTION: SC write out charter

##### Scope
* SC: In scope: Development of test suite and other software or services, feedback to specifications, data model/syntax for test software and test metadata.
* SC: Out of scope: ?

* MJ: Question about how we publish reports now. Do we need a separate discussion
* SC: Part of proposal 1 but if we need to discuss this, we can have another meeting as part of working through the issue and documenting this
* : We can have another discussion and add to the issues above

##### Deliverables
* SC: Charter will indicate "Test Suites" (or "Conformance Tooling" or whatever) but will not name specific tooling. Leave work items open.
* SC: The panel to continue with the work under Work Items listed under https://solidproject.org/TR/#work-item-test-suites as part of the Deliverable. New Work Items can be added provided need/agreement from the Panel and the Chair.
* YB: Should we also discuss conformance services - distinction between providing tooling versus providing an online service. It would be good to have this in scope
* SC: Yes, it is intended to be in scope (amended scope to include services)
* SC: Data/syntax for test suite design metadata.

#### PROPOSAL-6: Review Test Suite Panel Charter.
* SC: Anyone can review.
* SC: Requires TBL's approval.
* SC: ACTION for TBL to respond by 2022-06-07.
* SC: ACTION for SC and Test Suite Panel to update proposal based on any requested changes by 2022-06-10.

ACTION for TBL to respond by 2022-06-07 once charter is done.
ACTION for SC and Test Suite Panel to update proposal based on any requested changes by 2022-06-10.