slide-mode: Here π₯
test_single
profile testsWhile nf-test and pytest-workflow
(currently popular in nf-core) are both test frameworks for Nextflow pipelines
, they differ in structure, implementation, and features. Let's delve into these differences and explore why nf-test may offer some advantages over pytest-workflow
.
nf-test | pytest |
---|---|
Built on Groovy | Built on Python |
Specific to Nextflow | General-purpose testing framework for Python |
Provides specific assertions for testing Nextflow channels | Provides a range of assertions for testing Python code (or nextflow indirectly). Pytest assertions are not specific to any data structure or concept like channels in Nextflow |
NF-Core Modules pytest-workflow
main.nf
include { BCFTOOLS_SORT } from '../../../../../modules/nf-core/bcftools/sort/main.nf'
workflow test_bcftools_sort {
input = [
[ id:'test' ], // meta map
file(params.test_data['sarscov2']['illumina']['test_vcf'], checkIfExists: true)
]
BCFTOOLS_SORT ( input )
}
include { BCFTOOLS_REHEADER } from '../../../../../modules/nf-core/bcftools/reheader/main.nf'
include { BCFTOOLS_REHEADER as BCFTOOLS_REHEADER_VCF_GZ} from '../../../../../modules/nf-core/bcftools/reheader/main.nf'
include { BCFTOOLS_REHEADER as BCFTOOLS_REHEADER_BCF_GZ} from '../../../../../modules/nf-core/bcftools/reheader/main.nf'
nextflow.config
file process {
publishDir = { "${params.outdir}/${task.process.tokenize(':')[-1].tokenize('_')[0].toLowerCase()}" }
}
test.yml
file - name: bcftools sort test_bcftools_sort
command: nextflow run ./tests/modules/nf-core/bcftools/sort -entry test_bcftools_sort -c ./tests/config/nextflow.config -c ./tests/modules/nf-core/bcftools/sort/nextflow.config
tags:
- bcftools
- bcftools/sort
files:
- path: output/bcftools/test.vcf.gz
md5sum: 4f24467157f5c7a3b336481acf0c8a65
- path: output/bcftools/versions.yml
NF-Core Pipelines - Github CI Testing Framework
- name: Run pipeline with ${{ matrix.profile }} test profile
run: |
nextflow run ${GITHUB_WORKSPACE} -profile ${{ matrix.profile }},docker --outdir ./results
profile:
[
test_multi,
test_pacbio_its,
test_doubleprimers,
test_iontorrent,
test_single,
test_fasta,
test_reftaxcustom,
test_novaseq,
]
π΅ The local execution process can be challenging for newcomers due to the need to understand Python, interpret unconventional outputs, and handle container virtualization technologies.
PROFILE=docker pytest --tag fastqc_single_end --symlink --keep-workflow-wd
β οΈ There's a need to manually create and maintain md5sums. Even though this process can be scripted, it requires considerable effort, which can lead to potential neglect and decay over time.
π The creation of custom test logic is complex and requires advanced expertise, which could be a significant barrier for beginners.
when
closure, and a then
closure generate
Example - bcftools/sort/main.nf.test
nextflow_process {
name "Test Process BCFTOOLS_SORT"
script "modules/nf-core/bcftools/sort/main.nf"
process "BCFTOOLS_SORT"
config "tests/modules/nf-core/bcftools/sort/nextflow.config"
tag "bcftools_sort"
test("Sarscov2 Illumina VCF") {
when {
params {
outdir = "$outputDir"
}
process {
"""
input[0] = [
[ id:'test' ], // meta map
file(params.test_data['sarscov2']['illumina']['test_vcf'], checkIfExists: true)
]
"""
}
}
then {
assertAll(
{ assert process.success },
{ assert snapshot(path(params.outdir).list()).match() }
)
}
}
}
with
contains
assertContainsInAnyOrder
Allows you to make assertions about the contents of a channel without considering order.assert path(process.out.out_ch.get(0)).md5 == "64debea5017a035ddc67c0b51fa84b16"
assert path(process.out.out_ch.get(0)).json == path('./some.json').json
assert path(process.out.out_ch.get(0)).json.key == "value"
assert path(process.out.out_ch.get(0)).linesGzip.contains("Line Content")
def lines = path(process.out.gzip.get(0)).linesGzip[0..5]
assert lines.size() == 6
def lines = path(process.out.gzip.get(0)).grepLinesGzip(0,5)
assert lines.size() == 6
Snapshots are incredibly handy when you want to ensure your output channels or output files don't change unexpectedly
.
A typical snapshot test case creates a snapshot of the output channels or other objects, and then compares it to a reference snapshot file stored with the test (
*.nf.test.snap
) .
The test will fail if the two snapshots don't match. This indicates either an unexpected change, or the reference snapshot needs updating to reflect the new output of a process, workflow, pipeline, or function
.
The snapshot
keyword creates a snapshot of the object and its match method can then be used to check if its contains the expected data from the snap file.
assert snapshot(process.out).match()
assert snapshot(path("$outputDir/fastqc/test_fastqc.html")).match()
{
"Should run without failures": {
"content": [
"test_fastqc.html:md5,8455303cf8e02b1083e813bb2a35d99e"
],
"timestamp": "2023-05-22T23:22:36+0000"
}
}
Example of Snapshot Failure
Test Process FASTQC
Test [91188cdf] 'Sarscov2 Illumina Fastq' java.lang.RuntimeException: Different Snapshot:
Found:
[
"test_fastqc.html:md5,ec30463244068beb09abee14c95c55d4"
]
Expected:
[
"test_fastqc.html:md5,9759fe2ebad67decc1be1d80c8c0954d"
]
FAILED (16.021s)
Assertion failed:
1 of 2 assertions failed
When a snapshot test is failing due to an intentional implementation change, you can use the βupdate-snapshot flag to re-generate snapshots for all failed tests.
nf-test test tests/main.nf.test --update-snapshot
We will be generating tests for modules and pipeline of nf-core/ampliseq workflow
curl -fsSL https://code.askimed.com/install/nf-test | bash
# or
wget -qO- https://code.askimed.com/install/nf-test | bash
The init command set ups nf-test in the current directory.
nf-test init
The init command creates the following files: nf-test.config
and tests/nextflow.config
. It also creates a folder tests/
which is the home directory of your test code.
tests/nextflow.config
will come with config settings as below:
config {
testsDir "tests"
workDir ".nf-test"
configFile "tests/nextflow.config"
profile ""
}
Change configFile
from tests/nextflow.config
-> nextflow.config
to utilize pipeline's main config
nf-test generate process modules/nf-core/fastqc/main.nf
π nf-test 0.7.3
https://code.askimed.com/nf-test
(c) 2021 - 2023 Lukas Forer and Sebastian Schoenherr
Load source file '/workspace/ampliseq/modules/nf-core/fastqc/main.nf'
Wrote process test file '/workspace/ampliseq/tests/modules/nf-core/fastqc/main.nf.test
SUCCESS: Generated 1 test files.
tests/modules/nf-core/fastqc/main.nf.test
nextflow_process {
name "Test Process FASTQC"
script "modules/nf-core/fastqc/main.nf"
process "FASTQC"
test("Should run without failures") {
when {
params {
// define parameters here. Example:
// outdir = "tests/results"
}
process {
"""
// define inputs of the process here. Example:
// input[0] = file("test-file.txt")
"""
}
}
then {
assert process.success
assert snapshot(process.out).match()
}
}
}
name "Test Process FASTQC"
This is where you name your process.
script "modules/nf-core/fastqc/main.nf"
Here you specify the script you are testing.
process "FASTQC"
Here you indicate the process you are testing.
test("Should run without failures")
: This is the main test case and its description.
when
block: This is where you set up your test.
params
This is where you define parameters for your test. For example, you could define an output directory here.
process
This is where you define the inputs for the process you are testing and inputs are provided via index-position(input[0], input[1]).
then
block: This is where you assert the expected outcomes of your test.
assert process.success
Here you assert that the process should be successful.
assert snapshot(process.out).match()
Here you assert that the output of the process matches the snapshot stored in your test file.
To the test file:
when
blockthen
block:nextflow_process {
name "Test Process FASTQC"
script "modules/nf-core/fastqc/main.nf"
process "FASTQC"
profile "docker"
test("Sarscov2 Illumina Fastq") {
when {
params {
outdir = "$outputDir"
}
process {
"""
input[0] = [
[ id: 'test', single_end:true ],
[
file("https://raw.githubusercontent.com/nf-core/test-datasets/modules/data/genomics/sarscov2/illumina/fastq/test_1.fastq.gz", checkIfExists: true)
]
]
"""
}
}
then {
assertAll(
{ assert process.success },
{ assert snapshot(path("$outputDir/fastqc/test_fastqc.html")).match() }
)
}
}
}
Testing the module
nf-test test tests/modules/nf-core/fastqc/main.nf.test
The first time the test is run, the snapshot file is created.
Re-Run the test again to check if the snapshots match
output:
π nf-test 0.7.3
https://code.askimed.com/nf-test
(c) 2021 - 2023 Lukas Forer and Sebastian Schoenherr
Test Process FASTQC
Test [91188cdf] 'Sarscov2 Illumina Fastq' PASSED (19.82s)
SUCCESS: Executed 1 tests in 19.84s
nf-test generate pipeline main.nf
π nf-test 0.7.3
https://code.askimed.com/nf-test
(c) 2021 - 2023 Lukas Forer and Sebastian Schoenherr
Load source file '/workspace/ampliseq/main.nf'
Wrote pipeline test file '/workspace/ampliseq/tests/main.nf.test
SUCCESS: Generated 1 test files.
tests/main.nf.test
nextflow_pipeline {
name "Test Workflow main.nf"
script "main.nf"
test("Should run without failures") {
when {
params {
// define parameters here. Example:
// outdir = "tests/results"
}
}
then {
assert workflow.success
}
}
}
To the test file:
when
blockworkflow.success
or the followingnextflow_pipeline {
name "Test Workflow main.nf"
script "main.nf"
profile "test_single,docker"
tag "pipeline"
test("Illumina SE") {
when {
params {
outdir = "$outputDir"
}
}
then {
assertAll(
{ assert workflow.success },
{ assert snapshot(path("$outputDir/pipeline_info/software_versions.yml")).match("software_versions") },
{ assert snapshot(path("$outputDir/overall_summary.tsv")).match("overall_summary_tsv") },
{ assert snapshot(path("$outputDir/barrnap/rrna.arc.gff"),
path("$outputDir/barrnap/rrna.bac.gff"),
path("$outputDir/barrnap/rrna.euk.gff"),
path("$outputDir/barrnap/rrna.mito.gff")).match("barrnap") },
{ assert new File("$outputDir/barrnap/summary.tsv").exists() },
{ assert snapshot(path("$outputDir/cutadapt/cutadapt_summary.tsv")).match("cutadapt") },
{ assert new File("$outputDir/cutadapt/1a_S103_L001_R1_001.trimmed.cutadapt.log").exists() },
{ assert new File("$outputDir/cutadapt/1_S103_L001_R1_001.trimmed.cutadapt.log").exists() },
{ assert new File("$outputDir/cutadapt/2a_S115_L001_R1_001.trimmed.cutadapt.log").exists() },
{ assert new File("$outputDir/cutadapt/2_S115_L001_R1_001.trimmed.cutadapt.log").exists() },
{ assert new File("$outputDir/cutadapt/assignTaxonomy.cutadapt.log").exists() },
{ assert snapshot(path("$outputDir/dada2/ASV_seqs.fasta"),
path("$outputDir/dada2/ASV_table.tsv"),
path("$outputDir/dada2/ref_taxonomy.txt"),
path("$outputDir/dada2/DADA2_stats.tsv"),
path("$outputDir/dada2/DADA2_table.rds"),
path("$outputDir/dada2/DADA2_table.tsv")).match("dada2") },
{ assert new File("$outputDir/dada2/ASV_tax.tsv").exists() },
{ assert new File("$outputDir/dada2/ASV_tax_species.tsv").exists() },
{ assert new File("$outputDir/fastqc/1a_S103_L001_R1_001_fastqc.html").exists() },
{ assert new File("$outputDir/fastqc/1_S103_L001_R1_001_fastqc.html").exists() },
{ assert new File("$outputDir/fastqc/2a_S115_L001_R1_001_fastqc.html").exists() },
{ assert new File("$outputDir/fastqc/2_S115_L001_R1_001_fastqc.html").exists() },
{ assert snapshot(path("$outputDir/input/Samplesheet_single_end.tsv")).match("input") },
{ assert snapshot(path("$outputDir/multiqc/multiqc_data/multiqc_fastqc.txt"),
path("$outputDir/multiqc/multiqc_data/multiqc_general_stats.txt"),
path("$outputDir/multiqc/multiqc_data/multiqc_cutadapt.txt")).match("multiqc") }
)
}
}
}
nf-test test tests/main.nf.test
nf-test test tests/main.nf.test
π nf-test 0.7.3
https://code.askimed.com/nf-test
(c) 2021 - 2023 Lukas Forer and Sebastian Schoenherr
Test Workflow main.nf
Test [68e76581] 'Illumina SE' PASSED (261.213s)
SUCCESS: Executed 1 tests in 261.287s
nf-test test
π nf-test 0.7.3
https://code.askimed.com/nf-test
(c) 2021 - 2023 Lukas Forer and Sebastian Schoenherr
Found 1 files in test directory.
Test Process FASTQC
Test [91188cdf] 'Sarscov2 Illumina Fastq' PASSED (19.231s)
Test Workflow main.nf
Test [68e76581] 'Illumina SE' PASSED (149.098s)
SUCCESS: Executed 2 tests in 168.458s
* still evolving
Sarscov2 Illumina PE Sorted BAM
)process.success
assertionversions.yml
assert process.out.versions[0] ==~ ".*/versions.yml"
with(process.out.vcf[0]) {
assert get(0).id == "test"
assert get(1) ==~ ".*/test.vcf.gz"
}
assertAll(
{ assert process.success },
{ assert new File("$outputDir/bcftools/test.vcf.gz").exists() },
{ assert new File("$outputDir/bcftools/versions.yml").exists() }
)
assert snapshot(path(params.outdir).list()).match()
assert snapshot(path("$outputDir/bcftools/test.vcf.gz")).match()
assert new File("$outputDir/bcftools/versions.yml").exists()
assert snapshot(path("$outputDir/bcftools/test.vcf.gz").linesGzip[32-41]
def softwareVersions = path("$outputDir/pipeline_info/software_versions.yml").yaml
if (softwareVersions.containsKey("Workflow")) { softwareVersions.Workflow.remove("Nextflow") }
assert snapshot(softwareVersions).match("software_versions")
#nf-test
channel in nf-core Slack for further discussions