---
title: Adding unit tests to nf-core pipelines with nf-test
authors: James A. Fellows Yates & Nicolas Vannieuwkerke
tags: nf-test,nf-core,ci-testing,nextflow,tutorial
---

# Adding unit tests to nf-core pipelines with nf-test

## Introduction

We should really add unit tests, and pytest is nice but a lot of overhead.

Instead we will try with [nf-test](https://code.askimed.com/nf-test/).

This tutorial is based on experiences from the nf-test documentation and @nvnieuwk's testing.

## Install nf-test

I've installed this with conda into my personal nf-core conda environment (containing `nf-core` tools, and `nextflow`).

> TODO update with proper instructions

conda activate nf-core
conda install nf-test -c bioconda

## Pipeline repository update

Before we run any nf-test commands, we need to tweak the github repository slightly, by adding the default nf-test 'work' directory to our `.gitignore`

echo '.nf-test/' >> .gitignore

## Configure nf-test for your pipeline

First we need to generate a couple config files (`nf-test.config`, and `tests/nextflow.config`).

nf-test init

For nf-core pipelines we need to add `tests/nextflow.config` a couple of parameters as we do in any other nextflow config file.

params {
    outdir = '${params.outputDir}'
}

🛈 The purpose of using `params.outputDir` is to isolate all test files into a directory generated by nf-test`.nf-test/`. This also prevents test results files being uploaded to the git repostiory as specified in the [.gitignore](#pipeline-repository-update)

## Generate basic test template

Then we can make a template test file (ending in `.test`), by specifying we want to make a test for a nf-core workflow with a nf-test 'pipeline' template.

nf-test generate pipeline main.nf

> 🛈 workflows vs pipelines appear to have slightly different definitions of a workflow. A nf-test `workflow` corresponds to a nf-core `subworkflow`.

You can also rename the resulting `tests/main.nf.test` file to make it more specific, such as corresponding to a specific test profile.

## Writing a minimal test

Now you can edit the `tests/test.test.file`.

You should also remove the `when` block inside the generated test template file (for nf-core purposes, we will instead use our existing test `-profile`s to specify parameters).

You can also change the `then` block to `expect` to make it more clear (`nf-test` version 0.6.2 or higher).

For a very minimal example, you can just `expect` that the workflow should complete successfully, with `assert workflow.success`.

Therefore the test should look like as follows:

nextflow_pipeline {

    name "Test Workflow main.nf"
    script "main.nf"

    test("Should run without failures") {

        expect {
            assert workflow.success
        }

    }

}

## Running the minimal test locally

To run the test, you can then execute the `nf-test test` subcommand, pointing to your given test file, and then the particular profile you want to test against

> ⚠️ for `nf-test` we MUST specify a _double_ dash to profile (`--profile`)!

nf-test test tests/test_nothing.test --profile singularity,test_nothing

> 🛈 In this case, we are using the same profile name to that of the test, you could conceptually provide another `--profile`, assuming the output files are expected to be the same.

> 🛈 All tests are run by default in a

## Integrating the tests into nf-core GitHub Actions

To get this minimal test working in github actions, we need to make three changes to the nf-core `.github/workflows/ci.yml`

- Add a install nf-test step
- Update any parameter matrices you may have
- Update the test command

### Installing nf-test

To install the nf-test, add the following after the `nf-core/setup-nextflow` step.

      - name: Install Nextflow
        uses: nf-core/setup-nextflow@v1
        with:
          version: "${{ matrix.NXF_VER }}"

      - name: Install nf-test
        run: |
          sudo bash; mkdir /opt/nf-test; cd /opt/nf-test; wget https://github.com/askimed/nf-test/releases/download/v0.6.2/nf-test-0.6.2.tar.gz; tar xvfz nf-test-0.6.2.tar.gz; chmod +x nf-test;
          echo "/opt/nf-test" >> $GITHUB_PATH;

### Update parameter matrices

For updating the parameter matrices, you may have something like

        NXF_VER:
          - "21.10.3"
          - "latest-everything"
        parameters:
          - "--perform_longread_qc false"
          - "--perform_shortread_qc false"
          - "--shortread_qc_tool fastp"

You should move these into new test profiles, and replace the `parameters` matrix with one called `profiles`, and put the name of each test profile as entries, e.g. looking something like this:

        NXF_VER:
          - "21.10.3"
          - "latest-everything"
        profiles:
          - "test_longreadqcfalse"
          - "test_shortreadqcfalse"
          - "test_shortreadqctoolfastp"

### Update commands

Finally we can update the command itself, but replacing the `nextflow run` command with `nf-test test` and inserting our new profiles to the nf-test test file name and supply the profile itself.

So:

      - name: Run pipeline with test data
        with:
          command: nextflow run ${GITHUB_WORKSPACE} -profile test,docker --outdir ./results ${{ matrix.parameters }}

becomes:

      - name: Run pipeline with test data
        with:
          command: nf-test test ${GITHUB_WORKSPACE}/tests/${{ matrix.profiles }}.test --profile ${{ matrix.profiles }},docker

### Checking for output files

To convert the tests to _actual_ unit tests (i.e. checking output files), we can then update the `expect` block of each test file to specify things such as expected files, md5sums, string contents etc.

> 💡 One of the benefits of nf-test is you can write your own customs assertions in Groovy to check for anything you want.

#### Snapshot

nf-test provides a semi automated way of checking for file differences between CI tests using `snapshots`. This is somewhat equivalent to the `nf-core/tools` repeat test functionality that generates md5sum for you, but built directly into the testing tool.

You can supply any nextflow object (file/path, channel name, trace etc.) to the `snapshot()` function. The first time you run `nf-test test`, it will pick up any snapshot functions, and generate a new file with the same name as the test file but with `.snap` appended to it, that contains md5sums of each file.

Every subsequent test run will then compare the md5sum in the run against those listed in the `.snap` file.

        expect {
            assert workflow.success

            assert snapshot(path("${outputDir}/pipeline_info/database_sheet.valid.csv")).match()
        }

> ⚠️ Make sure to check the contents of each file looks correct before adding it the list of files for `snapshot`!

> ⚠️ Make sure to re-generate the 'snapshot' file if you expect contents of files to change! You can do this including `--update-snapshot` in your test command or by deleting the `.snap` file.

However in some cases, you will not be able to use snapshots due to variabilty of file contents. In this case we can 'manually' specify other types of files.

#### Manual Specifications

As with `pytestWorkflows`, we generally prefer `md5sum` checks to ensure there is no variability in the output (for reproducibility) via `.md5`, however in cases where a tool produces inheriently different output (e.g. rows in a different order in a table), you can check for strings with `.contains()`

        expect {
            assert workflow.success

            // File has a md5sum matching - preferred
            assert path("${outputDir}/fastqc/raw/2612_ERR5766176_B_raw_1_fastqc.html").md5 == "09d0dd1ef6219b5726b11d691fd22a37"
            
            // File contains the following string - backup e.g., when timestamp or variable row order 
            assert path("${outputDir}/fastqc/raw/2612_ERR5766176_B_raw_2_fastqc.html").readLines().join('').matches(".*FastQC Report.*")
            
            // File exists - last resort
            assert path("${outputDir}/multiqc/multiqc_report.html").exists()

        }

> 🛈 `outputDir` here corresponds to whatever is specified to `--outdir` ([default](https://code.askimed.com/nf-test/testcases/global_variables/#outputdir) being a directory named 'output')

### Automating the generation of the entire file

<!-- Not really automated, but run your test profile, then use XYZ command to generate a list of files found within the directory, saving them into a file then use `sed` to replace with the necessary function -->