Some benefits of using a testing framework

• Version control and collaboration
  • git bisect
• Reproducibility
• Run tests locally

Anatomy of a pytest-workflow test

- name: Touch a file
  command: touch test.file
  files:
    - path: test.file

Example: gunzip test.yml file

- name: gunzip
  command: nextflow run ./tests/software/gunzip -entry test_gunzip -c tests/config/nextflow.config
  tags:
      - gunzip
  files:
      - path: ./output/gunzip/test_1.fastq
        md5sum: 4161df271f9bfcd25d5845a1e220dbec

nf-core modules create-test-yml

• Automates the creation of the yaml file(And the md5sums!)
• Runs the tests
• Just write the tests!

nf-core modules create-test-yml Example

include { STAR_GENOMEGENERATE } from '../../../../software/star/genomegenerate/main.nf' addParams( options: [args: '--genomeSAindexNbases 9'] )
include { STAR_ALIGN          } from '../../../../software/star/align/main.nf'          addParams( options: [args: '--readFilesCommand zcat'] )

workflow test_star_alignment_single_end {
    input = [ [ id:'test', single_end:true ], // meta map
              [ file("${launchDir}/tests/data/generic/fastq/test_single_end.fastq.gz", checkIfExists: true) ]
            ]
    fasta = file("${launchDir}/tests/data/generic/fasta/GCF_000019425.1_ASM1942v1_genomic.fna", checkIfExists: true)
    gtf   = file("${launchDir}/tests/data/generic/gtf/GCF_000019425.1_ASM1942v1_genomic.gtf", checkIfExists: true)

    STAR_GENOMEGENERATE ( fasta, gtf )
    STAR_ALIGN ( input, STAR_GENOMEGENERATE.out.index, gtf )
}

$ nf-core modules create-test-yml

                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 1.13


INFO     Press enter to use default values (shown in brackets) or type your own responses
? Tool name: star/align
Test YAML output path (- for stdout) (tests/software/star/align/test.yml):
File exists! 'tests/software/star/align/test.yml' Overwrite? [y/n]: y
INFO     Looking for test workflow entry points: 'tests/software/star/align/main.nf'
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
INFO     Building test meta for entry point 'test_star_alignment_single_end'
Test name (star align test_star_alignment_single_end):
Test command (nextflow run tests/software/star/align -entry test_star_alignment_single_end -c tests/config/nextflow.config):
Test tags (comma separated) (star_alignment_single_end,star_align,star):
Test output folder with results (leave blank to run test):
? Choose software profile  Docker
INFO     Running 'star/align' test with command:
         nextflow run tests/software/star/align -entry test_star_alignment_single_end -c tests/config/nextflow.config --outdir
         /var/folders/bq/451scswn2dn4npxhf_28lyt40000gn/T/tmp_p22f8bg
INFO     Test workflow finished!
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
INFO     Building test meta for entry point 'test_star_alignment_paired_end'
...
...

The pieces of the CI

• dorny/paths-filter GitHub action checks for changes
• Based on the changes actions creates a matrix of jobs
• Tests are ran on singularity, docker, and conda
• pytest-workflow & Linting gets ran against the tags
• On failures the logs get uploaded

Checking for Changes

- uses: dorny/paths-filter@v2
  id: filter
  with:
    filters: "tests/config/pytest_software.yml"

modules/tests/config/pytest_software.yml`

star/align:
  - software/star/align/**
  - tests/software/star/align/**

The matrix

matrix:
    nxf_version: ['21.04.0']
    tags: ['${{ fromJson(needs.changes.outputs.modules) }}']
    profile: ['docker', 'singularity', 'conda']

Running the tests

- name: Run pytest-workflow
    # only use one thread for pytest-workflow to avoid race condition on conda cache.
    run: TMPDIR=~ PROFILE=${{ matrix.profile }} pytest --tag ${{ matrix.tags }} --symlink --kwdof

Logs of the Fallen

- name: Upload logs on failure
  if: failure()
  uses: actions/upload-artifact@v2
  with:
      name: logs-${{ matrix.tags }}-${{ matrix.profile }}-${{ matrix.nxf_version }}
      path: |
          /home/runner/pytest_workflow_*/*/.nextflow.log
          /home/runner/pytest_workflow_*/*/log.out
          /home/runner/pytest_workflow_*/*/log.err
          /home/runner/pytest_workflow_*/*/work

Linting the modules

- name: Lint ${{ matrix.tags }}
  run: nf-core modules lint . -t ${{ matrix.tags }}

Running pytest-workflow

nf-core/modules $ conda install pytest-workflow
nf-core/modules $ PROFILE=docker pytest --tag gunzip --symlink
nf-core/modules $ PROFILE=docker pytest --tag gunzip --symlink --basetemp ~/scratch/tmp/pytest

Oh the Things We'll Test!

• Verify expected outputs and their integrity
• The CI can be generalized
• Only testing sections of pipelines that have changes
  • Save the GitHub runners!
• Finer grained testing of subworkflows

Examples PRs

• sarek
• rnaseq

End-2-End(e2e) Example

# NOTE For now these tests don't support any containers beside singularity
- name: Run default pipeline with test data
  command: nextflow run main.nf -profile test,docker
  tags:
    - default
  files:
    # fastqc
    - path: results/fastqc/RAP1_IAA_30M_REP1_1_fastqc.html
    - path: results/fastqc/RAP1_IAA_30M_REP1_1_fastqc.zip
    ...
    # multiqc
    - path: results/multiqc/star_salmon/multiqc_report.html
    ...

Testing a Subworkflow

include { MARKDUPLICATES } from '../../../../subworkflows/nf-core/markduplicates' addParams(
    markduplicates_options:      modules['markduplicates'],
    markduplicatesspark_options: modules['markduplicatesspark']
)
 
workflow test_markduplicates {
    input = [[id: 'test'],
            [file(params.test_data['nf-core']['test_paired_end_sorted_bam'], checkIfExists: true)],
            [file(params.test_data['nf-core']['test_paired_end_sorted_bai'], checkIfExists: true)]]
 
    MARKDUPLICATES ( input, false, true )
}

Testing a Subworkflow

- name: subworkflow markduplicates
  command: nextflow run ./tests/subworkflows/nf-core/markduplicates/ -entry test_markduplicates -c tests/config/nextflow.config
  tags:
    - markduplicates
    - gatk4
  files:
   - path: output/preprocessing/test/markduplicates/test.md.bam
   - path: output/preprocessing/test/markduplicates/test.md.bam.bai

Testing a subworkflow

markduplicates:
  - modules/nf-core/software/gatk4/markduplicates/main.nf
  - subworkflow/local/markduplicates.nf
  - tests/subworkflow/local/markduplicates/**