---
# System prepended metadata

title: 'Fighting flaky tests: Test stability pipeline'

---

# Fighting flaky tests: Test stability pipeline

It is too expensive find out that the tests are flaky after they have been merged in and are already running in our CI/CD pipeline. What's more, if the deployment pipeline is stopped because of a flaky test, developers will slowly start to lose trust in the tests and with time they will ignore them completely. 

![](https://i.imgur.com/TBM3h2A.png)






## Sources of instability

There are numerous reasons for tests to become unstable: 


![](https://i.imgur.com/t9iq6Sg.png)







## Tackling instability: Test stability pipeline

In order to find out if our tests are stable, we need a setup that allows us to execute our tests N times against a stable environment. This can be accomplished with a very simple CI job which executes tests that you pass in manually. Nonetheless, the flow can be completely automated.

![](https://i.imgur.com/FiK6od0.png)


### Identifying new & modified tests 

Most of the modern test frameworks have built-in options to execute tests that have been modified when comparing against our master(main) branch. Few examples from [Jest](https://jestjs.io/): 



---

```
npm run test --onlyChanged
```

![](https://i.imgur.com/HtCnzma.png)




---

```
npm run test --changedSince origin/master
```
 

![](https://i.imgur.com/4S5qUyN.png)


If your test framework lacks this feature, you can utilise `git diff --name-only` by comparing new or modified files in your branch versus the master and then parsing out the test names from the list which you can pass to your test runner.


## Configuring your CI 

Lets create a simple pipeline that is going to use Jest's native functionality of identifying new and modified tests, few Jenkins parameters and a shell script to execute everything.


**package.json**

We are going to use --changedSince for identifying changes in our local branch vs origin/master.
```
"scripts": {
    "test:stability:pipeline": 
        "jest --passWithNoTests --changedSince origin/master"
}
```


**StabilityPipeline.groovy**


Adding a couple of parameters, the goold old TEST_URL and NUMBER_OF_EXECUTIONS that is going to default to 30.
```
parameters {
    string(defaultValue: '30', name: 'NUMBER_OF_EXECUTIONS')
    string(defaultValue: 'https://google.com', name: 'TEST_URL')
}
```



**Shell script**

And finally, a shell script to loop our tests for the amount provided in our build parameters.

```
for i in {1..${NUMBER_OF_EXECUTIONS}}; do npm run test:stability:pipeline" || exit 1; done"]`
```





## What's next?

* Configure your CI to run the stability pipeline whenever a pull request with new/modified tests is opened.  
* Include links to stability pipeline in your PR templates. 
* Run your historic tests in the stability pipeline

### Test stability service

A service that tracks the status of the tests and provides the list of stable tests to the test runner during the build time. Read more at [Test Stability Service](https://hackmd.io/@2YvFfTz8SLSEMD7Njahfhw/sda)


![](https://i.imgur.com/JU5Ie8j.png)


### Automatic unstaging of flaky tests

1. A test fails in our CI/CD pipeline
2. That test is sent to be executed N times in the stability pipeline
3. If the test does not fail 100% of the time, mark the test as unstable 

### Automatically create issues for unstable tests 

Unstable tests require work. Creating Jira's manually is so 2010. Create them automatically using your favorite issue trackers APIs. 

[JIRA REST API](https://developer.atlassian.com/server/jira/platform/jira-rest-api-examples/#creating-an-issue-using-a-project-key-and-field-names)

[Github REST API](https://docs.github.com/en/rest/reference/issues#create-an-issue)