# Solid Test Suite Panel 2021-03-01 https://meet.jit.si/solid-test-suite ## Present * Sarven * Michiel * Yvo * Pete * Maria D. * Jan S. * Alain ## Agenda * Announcements: * PRs: * Issues: * Why solidcommunity.net is not going through the test suite? * Why re-discuss the test suite given that it was complete? * Discussion: * TS https://solidproject.org/TR/#work-items and major/minor milestones * Communication and coordination plan * Test Suite coverage (required, optional) ## Minutes * Maria: 1) two issues 2) state of test suite - when can we ask for implementations? * Michiel: solidcommunity.net runs NSS - one of the tested implementations.. plan is to switch to CSS down the line. Using the live server to test. * Pete: re discussion on the completeness of the test suite wrt to the specifcification. perhaps the completion in December was optimistic? how confident are we about the tests? the current test suite doesn't/may not have the nuances. what do the scores mean/entail? have been looking at LDP test reports eg. required/optional features... suggest to increase the breadth of the tests.. and how we communicate the information about the tests. did some poc to see which platforms would be suitable - and part of the reason why the alternative approach came to be. * Emmet: discussion point: the way tests are written. for greatest number of people to understand the tests.. and spec people to validate the tests. imo: tests are written in declarative language... to keep the noise down and possibly be more accessible. when is it okay to test the server? eg. looking at a reports table (with scores) - what does the table communicate to different parties? sets a perception. eye of the beholder. getting full coverage is going important but need to be clear on the correctness of the tests. * Michiel: goes both ways.. if an implementation does more work / conforming to the spec, should get more points * Emmet: more about conformance then points. what process shall we take .. who/how do we agree/approve. * Michiel: anyone can publish their own test suites * Emmet: fundamentally disagree on particular entities creating their own test... we are talking about delivering certain quality. * Michiel: that may be an issue if we end up with two test suites - lets come back to this though. * Pete: test suite description document.. how do we link test suite description to the spec... and links to the tests that are run. that doc is used as input. test suite parses that and ... results are not pass/fail but specific on degree of conformance(?) * Michiel: i think we generally all agree on that.. main issue to changing how the suite is designed * Pete: -- tracing. jest is designed for usually well-known . but there are some gaps given the bigger picutre * Michiel: what aresome of the limtiations? * Pete: people implemented differen runners. set at run time what tests to run.. to turn on/off via env variables.. but ut have to do that for multiple dimensions. also looked into: can we get http requets/responses logged and combined with individual test cases.. for jest it seems to be no. * Michiel: I used debug output.. now it can output the curl -v. that's separate from the results. when you creat ethe report, it only includes pass/fail - reason as well. could put more info but not the whole transcript. not necessarily what you want - however full record can help (if you ) . Can you use Karate a DSL? * Pete: the actual tests are using "grawl"?-vm * Emmet: but that might be moving from declarative to imperative.. we should provide libs for those things.. to keep tests clean * Michiel: we tried several approaches (including above).. jest seemed to be the reasonable path forward - that's whwere we are now. * Emmet: the proposed lib/approach works out there. If I understand you correctly, .. * Michiel: would you be willing to do everything ou want to do in TypeScript.. * Pete: Was jest the framework? JS is not the issue.. it is the framework. the cost is coming from elsewhere. * Michiel: i get that that's less work in java(script).. but * Yvo: is there an upgrade path from current test suite to proposed? instead of starting from scratch. the current ts has limtiations.. but took quite a bit of work - alignment with existing implementations. need to do the same for anything new. it could be very difficult for the community. * Pete: the difficulty wasn't the Js.. but making the test what it had to .. and how it would validate the test are doing the right thing.. making sure it did the right sequence of things. there is a lot of value transferred from existing work. * Emmet: not so much work.. could dedicate more energy into - couple of weeks? * Yvo: would like links to the tests. must/should/may. switch on/off for certain implementations.. * Pete: description of the implementations would be useful - and use that for the tests eg. CRUD, w/ w/o authentication.. eg. CSS didn't support WAC-Allow until recently. was able to skip that test when experimenting.. and that should be accounted for. * Alain: a lot can be improved. the test suite is not meant for developing new servers.. if tests fail and not published is not a problem. the end results is when we reach milestones in a product.. if not achieved, nothing to publish. * Emmet: can you elaborate * Alain: the test suite need not include all the logic needed for development that might not be needed. * Emmet: it only knows what it nede sto know about the spec. different audiences for the test suite.. i agree with you on only testing * Sarven: difference between what the spec says and test setup * Michiel: (re tim) you want to be specific about the requirements.. don't see rewrite attractive at this time. but obviously not from scratch. if we move to java (fromt he current setup).. then people may be (at least one) less active. concern is that if we go in one direction that only one party can invest -- base don particular lang.. then it would be generally governed by that.. * Emmet: however, the idea is to work on different layers. * Michiel: if prog lang is not important, then lets not switch.. * Emmet: Java is not the important bit of this. more people would probably understand the declarative/gherkin type of tests * Michiel: we have a test suite.. active members.. fwiw, it is going well. lets not have two different suites managed by different groups * Emmet: alignment with everyone may be challenging * Michiel: can you rewrite some tests in Gherkin? * Emmet: can spec writers examine the tests? * Yvo: mashup? oidc+crud+wac .. jest with one, and another. we can ask the community what's easier. * Pete: Extending the test harness vertically (spec -> tests -> reports) .. * ## Actions * What would be the best way to verify the results? * Can we have a session to iron out what parts will be written by whom? *