{%hackmd theme-dark %}
# Hackathon March 2021
---
# DSL2
<!-- TODO optional: add images, gifs etc. -->
---
# Primary focus
<!-- TODO: State the goal of your group -->
1. Add new modules
2. Add tests to existing modules
3. Add overhead memory requirements
---
# Group members
<!-- TODO: add all group members -->
- Harshil Patel
- Kevin Menden
- Friederike Hanssen
- Gregor Sturm
- Jose Espinosa
- Michael Heuer
- Francesco Lescai
- Robert Petit
- Mark S. Hill
- Carlos Ruiz
- Batool Almarzouq
- Santiago Revale
- Aron Skaftason
- Hédia Tnani
- Anthony Fullam
- Suzanne Jin
- Ravneet Bhuller
- Kevin Brick
- Pernilla Ericsson
- Alex Peltzer
- Maxime Borry
- Yuk Kei Wan
- Nick Toda
- Edmund Miller
---
# New modules - Achievements
## Update test files for all modules
- New test_data.config
- Requires ALL modules to be updated
- Everyone involved
## New module sequenzautils/bam2seqz
- opened a new issue
- work in progress
## New module sequenzautils/gcwiggle
- PR submitted https://github.com/nf-core/modules/pull/345 on Tuesday
- all tests passed!
- work complete!
- reviewing complete! Many thanks to Harshil for the review!
- merged into https://github.com/nf-core/modules/pull/345#pullrequestreview-619327844
- PR closed on Wednesday!
## New module cnvkit
- merged into nf-core/modules https://github.com/nf-core/modules/pull/173
- many thanks to all the reviewers!
- PR closed on Tuesday!
## New module picard/collectwgsmetrics
- Answered issue https://github.com/nf-core/modules/issues/264
- Most of the heavy-lifting had already been done by [hpatel](@YO3oG0hdSw2Gv5IWHc0nfA)
- PR submitted: https://github.com/nf-core/modules/pull/304
- Currently failing 1 EditorConfig lint test
- Info: Can't use md5sum checks for this picard module with pytest due to time stampe in output. Used contain arguments instead.
- Info: Pytest still complains about exit code locally, even though running the nextflow command on it's own seems to work fine
## New module prokka
- Adds module for [Prokka](https://github.com/tseemann/prokka)
- Submitted by Robert (@rpetit3)
- Answered Issue https://github.com/nf-core/modules/issues/288
- PR Merged https://github.com/nf-core/modules/pull/298
- All the CI tests are passing
- First nf-core module PR by [rpetit3](@d6tmnqZgSaGbC-FcqZEFkg), so feed back appreciated
- Some outputs contain timestamps and MD5s could not be used
- Thanks for the reviews!
## Update module shovill
- Updates the deprecated module for [Shovill](https://github.com/tseemann/shovill)
- Submitted by Robert (@rpetit3)
- Answered Issue https://github.com/nf-core/modules/issues/329
- PR Submitted https://github.com/nf-core/modules/pull/337
- Included test for each assembler
- Had to use larger test data from [nf-core/test-datasets](https://github.com/nf-core/test-datasets/tree/bacass)
- Tests failing due to memory issue with shovill_spades: https://github.com/nf-core/modules/pull/337#issuecomment-805159617
- Set SPAdes to assemble with k=31 to avoid out of memory error
- Some MD5s are not reproducible for some outputs due to timestamps and contigs changing orders
## New module vcftools
- PR (https://github.com/nf-core/modules/pull/334) merged in to master
- Responds to issue https://github.com/nf-core/modules/issues/214
- Notes:
- Quite awkward getting something like vcftools to work well as a module, so solutions provided do not exhaustively cover all use-cases. But, they can be flexibly applied so individual users should be able to easily tailor to their own requirements.
## New module samtools/merge
- Added samtools merge, which merges multiple sorted bam files into one bam file
- Merged PR: https://github.com/nf-core/modules/pull/296
## New module samtools/fastq
- Added samtools fastq, which converts BAM file into FASTQ file
- Merged PR: https://github.com/nf-core/modules/pull/316
- Then modified the module in order to produce compressed fastq.gz files
- Submitted PR: https://github.com/nf-core/modules/pull/339
## New module strelka/germline
- Addapted code from Sarek to make it fit to modules
- Merged PR: https://github.com/nf-core/modules/pull/340
## New module ucsc/bed12tobigbed
- A part of the dsl2 modules used in the nanoseq dsl2 conversion (https://github.com/nf-core/nanoseq/issues/108)
- Submitted PR: https://github.com/nf-core/modules/pull/302
- Previous PR got conflicts
## New module freebayes/single
- Created the software
- Running the tests, error encountered
## New module fgbio/fastqtobam
- Added module FastqToBam from FGBIO toolkit, to be used as part of workflow to handle UMIs
- Responds to issue https://github.com/nf-core/modules/issues/189
- PR submitted as https://github.com/nf-core/modules/pull/306
- Tuesday update:
- quite some work into fixing a strange change in the pytest yaml, and mismatch with upstream (had to revert changes, rebase, checkout a specific file, add missing test manually)
- pytests are failing with different messages
- docker: complaining a file that should be there is not actually there. running test locally (but error message is not transparent with pytest) seems there might be a problem with the docker image of fgbio in biocontainers
- conda: test runs just fine locally, but on github CI pytest it complains the md5sum is not equal to expected
- singularity: test complains md5sum not equal to expected (same as with conda)
- Wednesday update
- after further testing (all nextflow local run worked, while pytest variable depending on environment) I have removed the md5sum for the output bam file
- now both conda and singularity CI tests pass
- further conflicts resolved
- docker remains an issue, with likely the biocontainer running into a library problem: this might require modification of the original container and therefore cannot be resolved within the timeframe of the hackathon.
## New module adam/markduplicates
- Ran into issue with ``-u $(id -u):$(id -g)`` specified in `docker.runOptions` failing tests; see discussion on Slack and e.g. https://github.com/nf-core/tools/pull/351#issuecomment-581320133
- https://github.com/nf-core/modules/pull/308
- See also https://github.com/nf-core/modules/pull/315
## New module unicycler
- Added module unicycler
- Responds to issue https://github.com/nf-core/modules/issues/290
- PR submitted as https://github.com/nf-core/modules/pull/307
- One of the missing modules for viralrecon DSL2 migration, see https://github.com/nf-core/viralrecon/issues/149
## New module: AdapterRemoval
- Adding adapterRemoval module
- Issue: https://github.com/nf-core/modules/issues/286
- PR: https://github.com/nf-core/modules/pull/309
## New module: bismark/align
- added module for alignment with bismark
## New module: bismark/report
- added module for bismark_report
- generates single-sample reports from bismark alignments
## New module: bismark/summary
- added module for bismark_summary report
- generates summary reports over many samples
## New module: allelecounter
- Added allelecounter module (@Anthony)
- Issue: https://github.com/nf-core/modules/issues/185
- PR: https://github.com/nf-core/modules/pull/313
## New module Kallisto/quant
- Adds module for [Kallisto/quant](https://github.com/nf-core/modules/pull/351)
- PR: https://github.com/nf-core/modules/pull/351
- Submitted by Batool (@BatoolMM)
- Answered Issue https://github.com/nf-core/modules/issues/359
- Also, attempted to fix the test path for salmon/index and salmon/quant in this PR https://github.com/nf-core/modules/pull/385/
## New module: DamageProfiler
- Added DamageProfiler module
- Issue: https://github.com/nf-core/modules/issues/285
- Progress here: https://github.com/maxibor/modules/tree/damageprofiler
- Probem with biocontainer (missing libfontconfig1 dep): need to update bioconda recipe
## New module: prodigal [Gregor]
- added prodigal module for gene prediction in procaryotes
- PR: https://github.com/nf-core/modules/pull/333
## New module: gatk4/applybqsr
- Submitted by Carlos Ruiz (@yocra3)
- Issue: https://github.com/nf-core/modules/issues/196
- PR: https://github.com/nf-core/modules/pull/331
- Merged with master branch
## New module: gatk4/baserecalibrator
- Submitted by Carlos Ruiz (@yocra3)
- Issue: https://github.com/nf-core/modules/issues/197
- PR: https://github.com/nf-core/modules/pull/327
- Awaiting review
## New module: gatk4/indexfeaturefile
- Issue: https://github.com/nf-core/modules/issues/310
- In progress (@santiagorevale)
## new module: gatk4/fastqtosam
- Added gatk4 module for converting fastq to sam files
- Merged with master branch
## New module: gatk4/genotypegvcfs
- Issue: https://github.com/nf-core/modules/issues/200
- In progress (@santiagorevale)
## New module: gatk4/markduplicates
- Submitted by John Juma (@ajodeh-juma)
- Issue: https://github.com/nf-core/modules/issues/202
- PR: https://github.com/nf-core/modules/pull/356
## New module: ensembl-VEP
- Issue: https://github.com/nf-core/modules/issues/215
- In progress (@HediaTnani)
## New module: msisensor/scan
- Added msisensor/scan module (@kevbrick)
- Issue: https://github.com/nf-core/modules/issues/207
- PR: https://github.com/nf-core/modules/pull/343
- This module was not in original issue but msisensor/msi requires output
- All checks have passed
- Awaiting review
## New module: msisensor/msi
- Added msisensor/msi module (@kevbrick)
- Issue: https://github.com/nf-core/modules/issues/207
- PR: https://github.com/nf-core/modules/pull/343
- All checks have passed
- Awaiting review
## New module: kb/ref
- Submitted by (@flowuenne)
- Issue: https://github.com/nf-core/modules/issues/358
- Module for creating a reference for scrnaseq pipeline processing with kb
- In progress:
- test data does not seem to work with indexing of kb ref
- Need human test data. Local test with only chr21 worked. Simply need a small file with few genes and transcripts and subset genomic fasta.
- Will wait for new human test data added by [@Friederike Hanssen ](@KuylkJ0AS0-limIJRUVsnw ) to creat PR
## New module: kb/count
- Submitted by (@flowuenne)
- Issue: https://github.com/nf-core/modules/issues/360
- Module for quantifying scRNA-seq data (fastq files) using kb-python
- In progress:
- For proper tests, will need the index created by kb/ref from new test data. Local test done with index of chromosome 21 work.
- Will wait for new human test data added by [@Friederike Hanssen ](@KuylkJ0AS0-limIJRUVsnw ) to creat PR
## New module: NanoPlot
- Yuk Kei (Wednesday): this module also uses sequencing_summary.txt in addition to viralcon's NanoPlot module, which only uses the fastq option
- Submitted PR: https://github.com/nf-core/modules/pull/364
- PyTest working locally but not on GitHub checks
# Modules Testing - Achievements
## Human data set
<!-- State what you achieved here -->
- Rike (Monday): Search for a suitable human data set
- Rike (Tuesday): Download data, convert to fastqs, determine UMI Adapter, start sarek
- Rike (Wednesday): Data is finally running now to generate all the indices, unfortunately not enough reads mapped to chr6, so chr22 it is for now
## Nanopore SARS-CoV2 test dataset
- Yuk Kei (Tuesday & Wednesday): created a 100-read SARS-CoV2 test dataset with single-read fast5 files, fastq, bam/bai, and sequencing_summary.txt
- Merged PR (fast5, fastq, and bam/bai): https://github.com/nf-core/modules/pull/344
- Submitted PR along with NanoPlot (sequencing_summary.txt): https://github.com/nf-core/modules/pull/364
## Changes to test data
- Add readgroup to existing test bam files for compatibility with gatk tools
- Add recalibration table (.table) for testing gatk tools.
- Updated fastp tests to deal with non-deterministic ouputs
## Remove runOptions from docker profile
- https://github.com/nf-core/modules/pull/315
- Need to investigate further whether it is strictly necessary to use ``-u $(id -u):$(id -g)`` for Docker in nf-core
# Other - Achievements
* Re-organised test data for different platforms
* New test data config (@everyone!🎉)
* Update CI tests to add linting and to bypass limit (Edmund)
* Added more docs about local installation (properly tested modules has helped!)
* Updated PR template to encourage linking the corresponding issues [[#312 by Gregor](https://github.com/nf-core/modules/pull/312)]
* Add docs for running tests locally with pytest [[#338 by Harshil](https://github.com/nf-core/modules/pull/338)]
## modules with custom scripts
- Trying to find a way to include custom scripts in a module
- `${moduleDir}/bin/my_script.sh` works in principle but breaks the cache
- Short term: workaround with `#tools`?
- Proof-of-concept PR at https://github.com/nf-core/modules/pull/368
- Issue at https://github.com/nf-core/modules/issues/294 and open for further discussion
- Long term: Find nextflow solution: https://github.com/nextflow-io/nextflow/issues/1798
---
{"metaMigratedAt":"2023-06-15T20:39:06.729Z","metaMigratedFrom":"YAML","title":"DSL2","breaks":true,"description":"View the slide with \"Slide Mode\".","contributors":"[{\"id\":\"35a34875-3971-4e5c-ac89-b0a2d4098367\",\"add\":16,\"del\":0},{\"id\":\"5173dade-c6de-469a-b5b6-1c23f764a44e\",\"add\":129,\"del\":0},{\"id\":\"77ab669e-a660-49a1-9b0b-e15ca9910592\",\"add\":1130,\"del\":60},{\"id\":\"60ede81b-485d-4b0d-86bf-92161dcd277c\",\"add\":240,\"del\":0},{\"id\":\"d4f3c6d2-9b64-4de5-bfa9-87d1cb94b8d9\",\"add\":778,\"del\":370},{\"id\":\"7884452c-d0ba-43f3-80d5-b5b590a0848f\",\"add\":1070,\"del\":1},{\"id\":\"07755eb0-739a-4628-85e2-ad4a4a341d04\",\"add\":15,\"del\":0},{\"id\":\"688a7413-4fd0-4c99-9ff6-727407ceed7e\",\"add\":520,\"del\":2},{\"id\":\"2aeca590-9d00-4b4f-a58a-620945456c9f\",\"add\":960,\"del\":184},{\"id\":\"7910da1c-5445-47af-a650-e5d7751e0ccb\",\"add\":732,\"del\":38},{\"id\":\"363fd4d9-ec06-4815-a82c-bbb49e7263f7\",\"add\":1735,\"del\":82},{\"id\":\"edfde7d2-913c-44cb-9ed0-595b4217c807\",\"add\":1429,\"del\":0},{\"id\":\"1e801949-a5d2-4c45-809f-5d8d4cb8bf61\",\"add\":20,\"del\":0},{\"id\":\"18075d75-f877-4b51-b17a-0cffcae637e8\",\"add\":530,\"del\":0},{\"id\":\"b3e26486-ba5a-432a-b344-b66269ef9aed\",\"add\":1014,\"del\":574},{\"id\":\"88579839-24d2-42f9-88cd-f2d6f6275f3d\",\"add\":43,\"del\":46},{\"id\":\"58d7366a-fc00-4c36-8f10-9242384379fb\",\"add\":219,\"del\":24},{\"id\":\"06c7e49f-af37-4f6b-b094-b18161fcbb44\",\"add\":1204,\"del\":670},{\"id\":\"f15a127e-9209-4f6f-8fba-ade02f32a88a\",\"add\":161,\"del\":0},{\"id\":\"f3e155e2-bd62-40df-86a6-7182ef435db0\",\"add\":125,\"del\":0},{\"id\":\"665dffc1-f062-4af3-b3ed-82729104fe48\",\"add\":298,\"del\":0},{\"id\":\"5330f88d-e739-4bf1-81e0-9271ad784385\",\"add\":185,\"del\":3},{\"id\":\"7fa84593-65d5-4968-b597-725e4806c56e\",\"add\":267,\"del\":0},{\"id\":\"dc580f85-324a-4b69-9c14-3d203e9f1488\",\"add\":411,\"del\":3},{\"id\":\"431768db-2e23-4511-b354-281f7b383cd0\",\"add\":579,\"del\":0},{\"id\":\"b8b4ab64-7246-4f84-84c8-a57a8bf78983\",\"add\":865,\"del\":299},{\"id\":\"847ef72c-7712-4848-a58a-795a183c718a\",\"add\":323,\"del\":0},{\"id\":\"724deb57-8e98-4e6b-a641-b9a59333d929\",\"add\":94,\"del\":0},{\"id\":\"41476d21-4b95-4fee-8577-7df5bb3e16ba\",\"add\":23,\"del\":0}]"}