Hackathon March 2021

DSL2

Primary focus

Add new modules
Add tests to existing modules
Add overhead memory requirements

Group members

Harshil Patel
Kevin Menden
Friederike Hanssen
Gregor Sturm
Jose Espinosa
Michael Heuer
Francesco Lescai
Robert Petit
Mark S. Hill
Carlos Ruiz
Batool Almarzouq
Santiago Revale
Aron Skaftason
Hédia Tnani
Anthony Fullam
Suzanne Jin
Ravneet Bhuller
Kevin Brick
Pernilla Ericsson
Alex Peltzer
Maxime Borry
Yuk Kei Wan
Nick Toda
Edmund Miller

New modules - Achievements

Update test files for all modules

New test_data.config
Requires ALL modules to be updated
Everyone involved

New module sequenzautils/bam2seqz

opened a new issue
work in progress

New module sequenzautils/gcwiggle

PR submitted https://github.com/nf-core/modules/pull/345 on Tuesday
all tests passed!
work complete!
reviewing complete! Many thanks to Harshil for the review!
merged into https://github.com/nf-core/modules/pull/345#pullrequestreview-619327844
PR closed on Wednesday!

New module cnvkit

merged into nf-core/modules https://github.com/nf-core/modules/pull/173
many thanks to all the reviewers!
PR closed on Tuesday!

New module picard/collectwgsmetrics

Answered issue https://github.com/nf-core/modules/issues/264
- Most of the heavy-lifting had already been done by hpatel
PR submitted: https://github.com/nf-core/modules/pull/304
- Currently failing 1 EditorConfig lint test
- Info: Can't use md5sum checks for this picard module with pytest due to time stampe in output. Used contain arguments instead.
- Info: Pytest still complains about exit code locally, even though running the nextflow command on it's own seems to work fine

New module prokka

Adds module for Prokka
Submitted by Robert (@rpetit3)
Answered Issue https://github.com/nf-core/modules/issues/288
PR Merged https://github.com/nf-core/modules/pull/298
- All the CI tests are passing
- First nf-core module PR by rpetit3, so feed back appreciated
- Some outputs contain timestamps and MD5s could not be used
- Thanks for the reviews!

Update module shovill

Updates the deprecated module for Shovill
Submitted by Robert (@rpetit3)
Answered Issue https://github.com/nf-core/modules/issues/329
PR Submitted https://github.com/nf-core/modules/pull/337
- Included test for each assembler
  - Had to use larger test data from nf-core/test-datasets
  - Tests failing due to memory issue with shovill_spades: https://github.com/nf-core/modules/pull/337#issuecomment-805159617
    - Set SPAdes to assemble with k=31 to avoid out of memory error
- Some MD5s are not reproducible for some outputs due to timestamps and contigs changing orders

New module vcftools

PR (https://github.com/nf-core/modules/pull/334) merged in to master
Responds to issue https://github.com/nf-core/modules/issues/214
Notes:
- Quite awkward getting something like vcftools to work well as a module, so solutions provided do not exhaustively cover all use-cases. But, they can be flexibly applied so individual users should be able to easily tailor to their own requirements.

New module samtools/merge

Added samtools merge, which merges multiple sorted bam files into one bam file
Merged PR: https://github.com/nf-core/modules/pull/296

New module samtools/fastq

Added samtools fastq, which converts BAM file into FASTQ file
Merged PR: https://github.com/nf-core/modules/pull/316
Then modified the module in order to produce compressed fastq.gz files
Submitted PR: https://github.com/nf-core/modules/pull/339

New module strelka/germline

Addapted code from Sarek to make it fit to modules
Merged PR: https://github.com/nf-core/modules/pull/340

New module ucsc/bed12tobigbed

A part of the dsl2 modules used in the nanoseq dsl2 conversion (https://github.com/nf-core/nanoseq/issues/108)
Submitted PR: https://github.com/nf-core/modules/pull/302
Previous PR got conflicts

New module freebayes/single

Created the software
Running the tests, error encountered

New module fgbio/fastqtobam

Added module FastqToBam from FGBIO toolkit, to be used as part of workflow to handle UMIs
Responds to issue https://github.com/nf-core/modules/issues/189
PR submitted as https://github.com/nf-core/modules/pull/306
Tuesday update:
- quite some work into fixing a strange change in the pytest yaml, and mismatch with upstream (had to revert changes, rebase, checkout a specific file, add missing test manually)
- pytests are failing with different messages
- docker: complaining a file that should be there is not actually there. running test locally (but error message is not transparent with pytest) seems there might be a problem with the docker image of fgbio in biocontainers
- conda: test runs just fine locally, but on github CI pytest it complains the md5sum is not equal to expected
- singularity: test complains md5sum not equal to expected (same as with conda)
Wednesday update
- after further testing (all nextflow local run worked, while pytest variable depending on environment) I have removed the md5sum for the output bam file
- now both conda and singularity CI tests pass
- further conflicts resolved
docker remains an issue, with likely the biocontainer running into a library problem: this might require modification of the original container and therefore cannot be resolved within the timeframe of the hackathon.

New module adam/markduplicates

Ran into issue with -u $(id -u):$(id -g) specified in docker.runOptions failing tests; see discussion on Slack and e.g. https://github.com/nf-core/tools/pull/351#issuecomment-581320133
https://github.com/nf-core/modules/pull/308
See also https://github.com/nf-core/modules/pull/315

New module unicycler

Added module unicycler
Responds to issue https://github.com/nf-core/modules/issues/290
PR submitted as https://github.com/nf-core/modules/pull/307
One of the missing modules for viralrecon DSL2 migration, see https://github.com/nf-core/viralrecon/issues/149

New module: AdapterRemoval

Adding adapterRemoval module
Issue: https://github.com/nf-core/modules/issues/286
PR: https://github.com/nf-core/modules/pull/309

New module: bismark/align

added module for alignment with bismark

New module: bismark/report

added module for bismark_report
- generates single-sample reports from bismark alignments

New module: bismark/summary

added module for bismark_summary report
- generates summary reports over many samples

New module: allelecounter

Added allelecounter module (@Anthony)
Issue: https://github.com/nf-core/modules/issues/185
PR: https://github.com/nf-core/modules/pull/313

New module Kallisto/quant

Adds module for Kallisto/quant
PR: https://github.com/nf-core/modules/pull/351
Submitted by Batool (@BatoolMM)
Answered Issue https://github.com/nf-core/modules/issues/359
Also, attempted to fix the test path for salmon/index and salmon/quant in this PR https://github.com/nf-core/modules/pull/385/

New module: DamageProfiler

Added DamageProfiler module
Issue: https://github.com/nf-core/modules/issues/285
Progress here: https://github.com/maxibor/modules/tree/damageprofiler
Probem with biocontainer (missing libfontconfig1 dep): need to update bioconda recipe

New module: prodigal [Gregor]

added prodigal module for gene prediction in procaryotes
PR: https://github.com/nf-core/modules/pull/333

New module: gatk4/applybqsr

Submitted by Carlos Ruiz (@yocra3)
Issue: https://github.com/nf-core/modules/issues/196
PR: https://github.com/nf-core/modules/pull/331
Merged with master branch

New module: gatk4/baserecalibrator

Submitted by Carlos Ruiz (@yocra3)
Issue: https://github.com/nf-core/modules/issues/197
PR: https://github.com/nf-core/modules/pull/327
Awaiting review

New module: gatk4/indexfeaturefile

Issue: https://github.com/nf-core/modules/issues/310
In progress (@santiagorevale)

new module: gatk4/fastqtosam

Added gatk4 module for converting fastq to sam files
Merged with master branch

New module: gatk4/genotypegvcfs

Issue: https://github.com/nf-core/modules/issues/200
In progress (@santiagorevale)

New module: gatk4/markduplicates

Submitted by John Juma (@ajodeh-juma)
Issue: https://github.com/nf-core/modules/issues/202
PR: https://github.com/nf-core/modules/pull/356

New module: ensembl-VEP

Issue: https://github.com/nf-core/modules/issues/215
In progress (@HediaTnani)

New module: msisensor/scan

Added msisensor/scan module (@kevbrick)
Issue: https://github.com/nf-core/modules/issues/207
PR: https://github.com/nf-core/modules/pull/343
This module was not in original issue but msisensor/msi requires output
All checks have passed
Awaiting review

New module: msisensor/msi

Added msisensor/msi module (@kevbrick)
Issue: https://github.com/nf-core/modules/issues/207
PR: https://github.com/nf-core/modules/pull/343
All checks have passed
Awaiting review

New module: kb/ref

Submitted by (@flowuenne)
Issue: https://github.com/nf-core/modules/issues/358
Module for creating a reference for scrnaseq pipeline processing with kb
In progress:
- test data does not seem to work with indexing of kb ref
- Need human test data. Local test with only chr21 worked. Simply need a small file with few genes and transcripts and subset genomic fasta.
- Will wait for new human test data added by @Friederike Hanssen to creat PR

New module: kb/count

Submitted by (@flowuenne)
Issue: https://github.com/nf-core/modules/issues/360
Module for quantifying scRNA-seq data (fastq files) using kb-python
In progress:
- For proper tests, will need the index created by kb/ref from new test data. Local test done with index of chromosome 21 work.
- Will wait for new human test data added by @Friederike Hanssen to creat PR

New module: NanoPlot

Yuk Kei (Wednesday): this module also uses sequencing_summary.txt in addition to viralcon's NanoPlot module, which only uses the fastq option
Submitted PR: https://github.com/nf-core/modules/pull/364
PyTest working locally but not on GitHub checks

Modules Testing - Achievements

Human data set

Rike (Monday): Search for a suitable human data set
Rike (Tuesday): Download data, convert to fastqs, determine UMI Adapter, start sarek
Rike (Wednesday): Data is finally running now to generate all the indices, unfortunately not enough reads mapped to chr6, so chr22 it is for now

Nanopore SARS-CoV2 test dataset

Yuk Kei (Tuesday & Wednesday): created a 100-read SARS-CoV2 test dataset with single-read fast5 files, fastq, bam/bai, and sequencing_summary.txt
Merged PR (fast5, fastq, and bam/bai): https://github.com/nf-core/modules/pull/344
Submitted PR along with NanoPlot (sequencing_summary.txt): https://github.com/nf-core/modules/pull/364

Changes to test data

Add readgroup to existing test bam files for compatibility with gatk tools
Add recalibration table (.table) for testing gatk tools.
Updated fastp tests to deal with non-deterministic ouputs

Remove runOptions from docker profile

https://github.com/nf-core/modules/pull/315
Need to investigate further whether it is strictly necessary to use -u $(id -u):$(id -g) for Docker in nf-core

Other - Achievements

Re-organised test data for different platforms
New test data config (@everyone!🎉)
Update CI tests to add linting and to bypass limit (Edmund)
Added more docs about local installation (properly tested modules has helped!)
Updated PR template to encourage linking the corresponding issues [#312 by Gregor]
Add docs for running tests locally with pytest [#338 by Harshil]

modules with custom scripts

Trying to find a way to include custom scripts in a module
${moduleDir}/bin/my_script.sh works in principle but breaks the cache
Short term: workaround with #tools?
Proof-of-concept PR at https://github.com/nf-core/modules/pull/368
Issue at https://github.com/nf-core/modules/issues/294 and open for further discussion
Long term: Find nextflow solution: https://github.com/nextflow-io/nextflow/issues/1798

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`	在筆記中貼入程式碼
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.