The image file may be corrupted
The server hosting the image is unavailable
The image path is incorrect
The image format is not supported

pytest-workflow

Edmund Miller /

Image Not Showing Possible Reasons

The image file may be corrupted
The server hosting the image is unavailable
The image path is incorrect
The image format is not supported

Learn More →

@E_Miller88 /

Image Not Showing Possible Reasons

The image file may be corrupted
The server hosting the image is unavailable
The image path is incorrect
The image format is not supported

Learn More →

@Emiller88
University of Texas at Dallas

Overview

pytest-workflow and nf-core modules
Behind the curtains of the CI
Running pytest-workflow on nf-core modules locally
Testing pipelines with pytest-workflow

pytest-workflow

Writing workflows is hard
Testing if they are correct is even harder.
Testing with bash scripts or other code has some flaws.
Is this bug in the pipeline or in my test-framework?

Pytest-workflow aims to make testing as simple as possible so you can focus on debugging your pipeline.

pytest-workflow and nf-core modules

Some benefits of using a testing framework

Version control and collaboration
- git bisect
Reproducibility
Run tests locally

Anatomy of a pytest-workflow test

- name: Touch a file
  command: touch test.file
  files:
    - path: test.file

Example: gunzip `test.yml` file

- name: gunzip
  command: nextflow run ./tests/software/gunzip -entry test_gunzip -c tests/config/nextflow.config
  tags:
      - gunzip
  files:
      - path: ./output/gunzip/test_1.fastq
        md5sum: 4161df271f9bfcd25d5845a1e220dbec

`nf-core modules create-test-yml`

Automates the creation of the yaml file(And the md5sums!)
Runs the tests
Just write the tests!

`nf-core modules create-test-yml` Example

include { STAR_GENOMEGENERATE } from '../../../../software/star/genomegenerate/main.nf' addParams( options: [args: '--genomeSAindexNbases 9'] )
include { STAR_ALIGN          } from '../../../../software/star/align/main.nf'          addParams( options: [args: '--readFilesCommand zcat'] )

workflow test_star_alignment_single_end {
    input = [ [ id:'test', single_end:true ], // meta map
              [ file("${launchDir}/tests/data/generic/fastq/test_single_end.fastq.gz", checkIfExists: true) ]
            ]
    fasta = file("${launchDir}/tests/data/generic/fasta/GCF_000019425.1_ASM1942v1_genomic.fna", checkIfExists: true)
    gtf   = file("${launchDir}/tests/data/generic/gtf/GCF_000019425.1_ASM1942v1_genomic.gtf", checkIfExists: true)

    STAR_GENOMEGENERATE ( fasta, gtf )
    STAR_ALIGN ( input, STAR_GENOMEGENERATE.out.index, gtf )
}

$ nf-core modules create-test-yml

                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 1.13


INFO     Press enter to use default values (shown in brackets) or type your own responses
? Tool name: star/align
Test YAML output path (- for stdout) (tests/software/star/align/test.yml):
File exists! 'tests/software/star/align/test.yml' Overwrite? [y/n]: y
INFO     Looking for test workflow entry points: 'tests/software/star/align/main.nf'
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
INFO     Building test meta for entry point 'test_star_alignment_single_end'
Test name (star align test_star_alignment_single_end):
Test command (nextflow run tests/software/star/align -entry test_star_alignment_single_end -c tests/config/nextflow.config):
Test tags (comma separated) (star_alignment_single_end,star_align,star):
Test output folder with results (leave blank to run test):
? Choose software profile  Docker
INFO     Running 'star/align' test with command:
         nextflow run tests/software/star/align -entry test_star_alignment_single_end -c tests/config/nextflow.config --outdir
         /var/folders/bq/451scswn2dn4npxhf_28lyt40000gn/T/tmp_p22f8bg
INFO     Test workflow finished!
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
INFO     Building test meta for entry point 'test_star_alignment_paired_end'
...
...

Behind the Curtains of the CI

The pieces of the CI

dorny/paths-filter GitHub action checks for changes
Based on the changes actions creates a matrix of jobs
Tests are ran on singularity, docker, and conda
pytest-workflow & Linting gets ran against the tags
On failures the logs get uploaded

Checking for Changes




- uses: dorny/paths-filter@v2
  id: filter
  with:
    filters: "tests/config/pytest_software.yml"

modules/tests/config/pytest_software.yml`



star/align:
  - software/star/align/**
  - tests/software/star/align/**

The matrix

Image Not Showing Possible Reasons
The image file may be corrupted
The server hosting the image is unavailable
The image path is incorrect
The image format is not supported
Learn More →




matrix:
    nxf_version: ['21.04.0']
    tags: ['${{ fromJson(needs.changes.outputs.modules) }}']
    profile: ['docker', 'singularity', 'conda']

The CI fails if matrix is larger than 256

Running the tests

- name: Run pytest-workflow
    # only use one thread for pytest-workflow to avoid race condition on conda cache.
    run: TMPDIR=~ PROFILE=${{ matrix.profile }} pytest --tag ${{ matrix.tags }} --symlink --kwdof

Logs of the Fallen

- name: Upload logs on failure
  if: failure()
  uses: actions/upload-artifact@v2
  with:
      name: logs-${{ matrix.tags }}-${{ matrix.profile }}-${{ matrix.nxf_version }}
      path: |
          /home/runner/pytest_workflow_*/*/.nextflow.log
          /home/runner/pytest_workflow_*/*/log.out
          /home/runner/pytest_workflow_*/*/log.err
          /home/runner/pytest_workflow_*/*/work

Linting the modules

- name: Lint ${{ matrix.tags }}
  run: nf-core modules lint . -t ${{ matrix.tags }}

Running pytest-workflow Locally

Running pytest-workflow

nf-core/modules $ conda install pytest-workflow
nf-core/modules $ PROFILE=docker pytest --tag gunzip --symlink
nf-core/modules $ PROFILE=docker pytest --tag gunzip --symlink --basetemp ~/scratch/tmp/pytest

Testing pipelines with pytest-workflow

Oh the Things We'll Test!

Verify expected outputs and their integrity
The CI can be generalized
Only testing sections of pipelines that have changes
- Save the GitHub runners!
Finer grained testing of subworkflows

Examples PRs

sarek
rnaseq

End-2-End(e2e) Example













# NOTE For now these tests don't support any containers beside singularity
- name: Run default pipeline with test data
  command: nextflow run main.nf -profile test,docker
  tags:
    - default
  files:
    # fastqc
    - path: results/fastqc/RAP1_IAA_30M_REP1_1_fastqc.html
    - path: results/fastqc/RAP1_IAA_30M_REP1_1_fastqc.zip
    ...
    # multiqc
    - path: results/multiqc/star_salmon/multiqc_report.html
    ...

Testing a Subworkflow












include { MARKDUPLICATES } from '../../../../subworkflows/nf-core/markduplicates' addParams(
    markduplicates_options:      modules['markduplicates'],
    markduplicatesspark_options: modules['markduplicatesspark']
)

workflow test_markduplicates {
    input = [[id: 'test'],
            [file(params.test_data['nf-core']['test_paired_end_sorted_bam'], checkIfExists: true)],
            [file(params.test_data['nf-core']['test_paired_end_sorted_bai'], checkIfExists: true)]]

    MARKDUPLICATES ( input, false, true )
}

Testing a Subworkflow

- name: subworkflow markduplicates
  command: nextflow run ./tests/subworkflows/nf-core/markduplicates/ -entry test_markduplicates -c tests/config/nextflow.config
  tags:
    - markduplicates
    - gatk4
  files:
   - path: output/preprocessing/test/markduplicates/test.md.bam
   - path: output/preprocessing/test/markduplicates/test.md.bam.bai

Testing a subworkflow

markduplicates:
  - modules/nf-core/software/gatk4/markduplicates/main.nf
  - subworkflow/local/markduplicates.nf
  - tests/subworkflow/local/markduplicates/**

Need help?

Repository: nf-core/modules
Tutorial: https://nf-co.re/usage/usage_tutorials
Chat: https://nf-co.re/join

Image Not Showing Possible Reasons

The image file may be corrupted
The server hosting the image is unavailable
The image path is incorrect
The image format is not supported

Learn More →

#channel

Follow nf-core on

Image Not Showing Possible Reasons

The image file may be corrupted
The server hosting the image is unavailable
The image path is incorrect
The image format is not supported

Learn More →

Image Not Showing Possible Reasons

The image file may be corrupted
The server hosting the image is unavailable
The image path is incorrect
The image format is not supported

Learn More →

Image Not Showing Possible Reasons

The image file may be corrupted
The server hosting the image is unavailable
The image path is incorrect
The image format is not supported

Learn More →

https://nf-co.re/

Image Not Showing Possible Reasons

The image file may be corrupted
The server hosting the image is unavailable
The image path is incorrect
The image format is not supported

Learn More →

Icons:
openmoji.org

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`	在筆記中貼入程式碼
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.

pytest-workflow

Overview

pytest-workflow

pytest-workflow and nf-core modules

Some benefits of using a testing framework

Anatomy of a pytest-workflow test

Example: gunzip test.yml file

nf-core modules create-test-yml

nf-core modules create-test-yml Example

Behind the Curtains of the CI

The pieces of the CI

Checking for Changes

modules/tests/config/pytest_software.yml`

The matrix Image Not Showing Possible Reasons The image file may be corruptedThe server hosting the image is unavailableThe image path is incorrectThe image format is not supported Learn More →

Running the tests

Logs of the Fallen

Linting the modules

Running pytest-workflow Locally

Running pytest-workflow

Testing pipelines with pytest-workflow

Oh the Things We'll Test!

Examples PRs

End-2-End(e2e) Example

Testing a Subworkflow

Testing a Subworkflow

Testing a subworkflow

Need help?

Example: gunzip `test.yml` file

`nf-core modules create-test-yml`

`nf-core modules create-test-yml` Example

The matrix

Image Not Showing Possible Reasons
The image file may be corrupted
The server hosting the image is unavailable
The image path is incorrect
The image format is not supported
Learn More →