---
# System prepended metadata

title: MTX Ideal Test Story
tags: [MTX]

---

# MTX Ideal Test Story
###### tags: `MTX`

## konveyor/crane
* Unit tests with coverage metric
-> Although more coverage is better than less, I actually am less concerned with
absolute coverage % and more interested in being able to 1) ensure that it does
not decrease and block PRs that do that and 2) be able to measure relative improvement
over time, and set goals for that. Can we increase our test coverage by X amount
by Y date.

* Short term integration tests (sanity tests)
-> Does a change break the most basic of use-cases?
Ex: Can the operator still install all the components?
Can we run a simple, small stateful migration from one cluster to another?
(If it's easier to start with a single cluster, let's crawl walk run)

* Unit tests + short term e2e should gate PRs
* Code coverage decrease should gate PRs

## crane-runner
-> Short term e2e tests here make sense to exercise the Tekton tasks that are
executing crane. Should focus on, do the pipelines we defined for MTX actually
run a simple stateful migration?

## MTX
Can we build a test suite that drives the UI and does a simple stateful migration?
Should build the simple stateful migration described above and gate PRs as a short term test.

## Nightly Suite
We should have a longer test suite that has a collection of scenarios that are
all run. When BZs are fixed, scenario is written to prove that bug was fixed.
Obviously this means it should be straightforward to author new test scenarios.

## Quality
We should measure the test flake rate in some way. Can we learn how often reported
failures are actual problems vs. not? Requires manual intervention and recording I think,
which is expensive.

Rotating test nanny amongst the team? Responsible for investigating nightly
failures and driving effort to fix the suite.

Build culture that when new code is contributed, it should be tested.

Nightly, longer running test suite should be in a shared place where both dev
and QE can contribute.
