# MTX Ideal Test Story ###### tags: `MTX` ## konveyor/crane * Unit tests with coverage metric -> Although more coverage is better than less, I actually am less concerned with absolute coverage % and more interested in being able to 1) ensure that it does not decrease and block PRs that do that and 2) be able to measure relative improvement over time, and set goals for that. Can we increase our test coverage by X amount by Y date. * Short term integration tests (sanity tests) -> Does a change break the most basic of use-cases? Ex: Can the operator still install all the components? Can we run a simple, small stateful migration from one cluster to another? (If it's easier to start with a single cluster, let's crawl walk run) * Unit tests + short term e2e should gate PRs * Code coverage decrease should gate PRs ## crane-runner -> Short term e2e tests here make sense to exercise the Tekton tasks that are executing crane. Should focus on, do the pipelines we defined for MTX actually run a simple stateful migration? ## MTX Can we build a test suite that drives the UI and does a simple stateful migration? Should build the simple stateful migration described above and gate PRs as a short term test. ## Nightly Suite We should have a longer test suite that has a collection of scenarios that are all run. When BZs are fixed, scenario is written to prove that bug was fixed. Obviously this means it should be straightforward to author new test scenarios. ## Quality We should measure the test flake rate in some way. Can we learn how often reported failures are actual problems vs. not? Requires manual intervention and recording I think, which is expensive. Rotating test nanny amongst the team? Responsible for investigating nightly failures and driving effort to fix the suite. Build culture that when new code is contributed, it should be tested. Nightly, longer running test suite should be in a shared place where both dev and QE can contribute.