Conformance tests/lint (Esteban's project #1)

--- robots: noindex, nofollow tags: pitch, internship, build --- [toc]  # Conformance tests/lint (Esteban's project #1) ## Problem  In a component library, it's important that the components follow consistent API patterns and coding style. This makes them both easier to use and easier to maintain. However, even with clear coding standards and careful reviewers, inconsistencies can easily slip by. For example, this might happen if someone copy-pastes a component and forgets to change the name, or different component authors come up with the a different name/signature for a prop that does the same thing. This is where conformance tests (and lint rules) come in. They run by default against all components and provide automated verification that standards are being followed. Fluent UI v0 has some convergence tests, but they're tightly coupled to the host project (not reusable in other projects or for partners) and may not quite match the standards we want for our new converged components. Another consideration is that some conformance tests could also be implemented as ESLint rules, which may be more efficient in some cases. ## Appetite  TODO - Flesh out project then determine timeline (probably 2-4 weeks) ## Solution  TODO: As part of defining the solution to this problem, you'll need to answer these questions: 1. What is our ideal state with component conformance? (define the set of rules we want) 2. What tools/resources exist to help us get there? - Within our repo (mainly v0 conformance tests) - Are there existing ESLint rules for any of the practices we'd like to enforce? - Are there any existing tools for component conformance testing (with Jest)? 3. What should be the dev experience running the tests? 4. What is the sequencing for bringing tests online? (Can't do it all at once, so what comes first?) ### More details/considerations #### Defining rules Defining the set of rules will require looking through the existing tests (see [Conformance tests](/g46wd9OtTzSAbq5d7QFwMA)) and determining which ones are relevant to converge, as well as talking to team members (David, Levi, others?) to determine what we want to test for new components. (v0 tests are a good starting point, but there are likely some other things we'll want to cover.) #### Implementing and sharing rules For rules which can be implemented with static code analysis, we should probably use ESLint. For the remaining rules/tests, we'll use Jest. In either case, the goal will be to set the rules up in a way that's reusable between packages (inside and maybe later outside our repo). Sharing ESLint rules is straightforward: we can make a new package `@fluentui/eslint-plugin-conformance` for custom rules (and custom config of existing rules), and other packages can depend on it. For Jest, we should first look online to see if someone else has written a tool for running component conformance tests. This seems like a common problem, so you'd *think* someone would have made a tool for it by this point. When evaluating any tools, we should consider: - Can it test the things we care about? - Does it have any assumptions baked in which will conflict with our preferences/style? - How hard is it to integrate? - If we find a tool we're interested in integrating, we'll set a hard limit on how long we can spend trying to get it working (to avoid situations like with Monaco last summer) If we do have to write our own conformance test runner, here's part of what I'd envision it looking like: - Separate package `@fluentui/react-conformance` or similar - Has a default set of conformance tests to run for a given component - Exports a function accepting: - either a glob of component files, or the path to a single component file (see TBD below) - override options (turn off certain tests for certain components) - possibly custom tests to run (optional) - TBD: how should this be run? - Call the function within each component's other tests, like v0 does today (downside: people could forget/decline to call it) - Call from a centralized conformance test file in each project (like `conformance.test.tsx`) with a glob of all component files + customizations - Run as a separate build step/tool #### Using rules Primary focus will probably be on getting the rules working against the converged components. If those aren't in a state where conformance tests are relevant yet, the focus will be v0 (since those tests are in a better starting state), ensuring that we split out anything specific to v0 from things that will be more generally relevant. Applying to v7 is a bonus. ## Risks (Rabbit holes)  We could easily spend a lot of time investigating and attempting to integrate 3rd-party tools. Set a specific timeline to mitigate this (if it's not done by then, stop). There may be some philosophical differences between teams: for example, default exports; number of exports per file; interface/type location; type vs interface for props. We can always make more conformance tests, so we have to prioritize and scope what to tackle now. ## Out of scope (No-gos)  Fixing anything but the most trivial issues with components is out of scope. (Just disable the particular test for that component instead.) We probably won't fully implement every conformance test idea we think of, or build a "perfect" conformance test suite. ## Appendix: Rules today + ideas See [Conformance tests](/g46wd9OtTzSAbq5d7QFwMA)