# RFC 0xxx: Protocol Test Suite ## Summary Explain how the Aries ecosystem proves interoperability of implementations of its RFCs. ## Motivation (explain difference between interop that's subjective, and interop that's objectively measured) ## Tutorial Things we need to explain: * What is the scope of the protocol test suite? (Does it exercise encryption envelope? decorators? which protocols? Does it test credential issuance and verification?) * How normative is the protocol test suite? (need to reference text in root README.md about how an RFC gets ACCEPTED status?) * Who maintains it? (everybody, I think, right?) * What's the relative versioning expectation? (Does the test suite precede or follow an RFC? What duty does an RFC update have to do a corresponding update of the suite?) * How are the test suite results versioned, so old results don't get used against new code? * What is the contract between the test suite and the agent under test? * What are the duties of an implementation developer when they want to put their code under test? * How are test suite results published in RFC impl sections and elsewhere? * Are some tests in the suite more "core" or more "important" than others? If so, how do controversies about the classification of tests get resolved? * How are tests named and grouped, and how to test names and groups correspond to test suite results? * Are all results from the test suite equally normative, or can anybody publish test results with equal authority? * What is the contract that all test writers (as opposed to all developers) * How do we express the intentions of an agent (where it makes a claim about which protocols it intends to implement)? * How do we express the notion that some tests are not worth running if other tests don't pass? * How often do we expect the test suite to be re-run, and to have its results published (and are those answers different)? How do we detect stale results? * How can the test suite be hooked up to a CI/CD pipeline? * If the test suite gives different results for the same codebase in different contexts, how do we represent that?