# Fighting flaky tests: Test stability pipeline

It is too expensive find out that the tests are flaky after they have been merged in and are already running in our CI/CD pipeline. What's more, if the deployment pipeline is stopped because of a flaky test, developers will slowly start to lose trust in the tests and with time they will ignore them completely.


## Sources of instability

There are numerous reasons for tests to become unstable:


## Tackling instability: Test stability pipeline

In order to find out if our tests are stable, we need a setup that allows us to execute our tests N times against a stable environment. This can be accomplished with a very simple CI job which executes tests that you pass in manually. Nonetheless, the flow can be completely automated.


### Identifying new & modified tests

Most of the modern test frameworks have built-in options to execute tests that have been modified when comparing against our master(main) branch. Few examples from [Jest](https://jestjs.io/):

---

npm run test --onlyChanged


---

npm run test --changedSince origin/master


If your test framework lacks this feature, you can utilise `git diff --name-only` by comparing new or modified files in your branch versus the master and then parsing out the test names from the list which you can pass to your test runner.

## Configuring your CI

Lets create a simple pipeline that is going to use Jest's native functionality of identifying new and modified tests, few Jenkins parameters and a shell script to execute everything.

**package.json**

We are going to use --changedSince for identifying changes in our local branch vs origin/master.

"scripts": {
    "test:stability:pipeline": 
        "jest --passWithNoTests --changedSince origin/master"
}

**StabilityPipeline.groovy**

Adding a couple of parameters, the goold old TEST_URL and NUMBER_OF_EXECUTIONS that is going to default to 30.

parameters {
    string(defaultValue: '30', name: 'NUMBER_OF_EXECUTIONS')
    string(defaultValue: 'https://google.com', name: 'TEST_URL')
}

**Shell script**

And finally, a shell script to loop our tests for the amount provided in our build parameters.

for i in {1..${NUMBER_OF_EXECUTIONS}}; do npm run test:stability:pipeline" || exit 1; done"]`

## What's next?

* Configure your CI to run the stability pipeline whenever a pull request with new/modified tests is opened.
* Include links to stability pipeline in your PR templates.
* Run your historic tests in the stability pipeline

### Test stability service

A service that tracks the status of the tests and provides the list of stable tests to the test runner during the build time. Read more at [Test Stability Service](https://hackmd.io/@2YvFfTz8SLSEMD7Njahfhw/sda)


### Automatic unstaging of flaky tests

1. A test fails in our CI/CD pipeline
2. That test is sent to be executed N times in the stability pipeline
3. If the test does not fail 100% of the time, mark the test as unstable

### Automatically create issues for unstable tests

Unstable tests require work. Creating Jira's manually is so 2010. Create them automatically using your favorite issue trackers APIs.

[JIRA REST API](https://developer.atlassian.com/server/jira/platform/jira-rest-api-examples/#creating-an-issue-using-a-project-key-and-field-names)

[Github REST API](https://docs.github.com/en/rest/reference/issues#create-an-issue)