---
title: Pipelines
tags: march-2021, nf-core, hackathon, talk
description: View the slide with "Slide Mode".
---
{%hackmd theme-dark %}


# Hackathon March 2021

---

# Pipelines group

<!-- TODO: State the goal of your group -->
* Contributing to existing pipelines
* Creating new pipelines
* Porting pipelines to DSL2
* Pipeline maintenance: e.g. template updates

---

# Group Members

<!-- TODO: add all group members -->

- Maxime Garcia
- Gisela Gabernet
- Alex Peltzer
- Susanna Marquez
- Daniel Straub
- Daniel Lundin
- Emelie Nilsson
- Leon Bichmann
- Gervaise Henry
- Friederike Hanssen
- Daniel Schreyer
- Simon Heumos @subwaystation
- Anthony Underwood
- Andrea Guarracino @AndreaGuarracino
- @avantonder
- Sangram @sk-sahu
- Patrick Hüther @phue
- Alexandre Gilardet
- Sabrina Krakau
- Barry Digby
- Thanh Le Viet
---

# Sarek 3.0 - Achievements

#### Monday
- Maxime: working on [#358](https://github.com/nf-core/sarek/pull/358)
- Many people: working on modules

#### Tuesday
- Maxime: still working on [#358](https://github.com/nf-core/sarek/pull/358)
- Maxime: about to integrate more modules

#### Wednesday
- Maxime: Finally merged [#358](https://github.com/nf-core/sarek/pull/358)
- Maxime: Starting to work on integrating the numerous modules created during the hackathon

---

# scRNAseq - Achievements

#### Monday
- Sangram/Alex: Updated JSON schema + opened PR with latest template 1.13.1, Gisela reviewed it ;-)

#### Tuesday
- Sangram fixed container issues, PR merged to dev.
- Waiting for adding latest tools template and will make a minor release.

#### Wednesday
- Sangram - Taking care of PR with latest template 1.13.2, review helped by - Gisela :)
- Minor release v1.1.0 before porting into DSL2 - happening
- Adding modules for kallisto and kd (@FloWuenne, @Batool, @ggabernet)

---

# Hlatyping - Achievements

#### Monday
Please chat on #hlatyping in Slack if possible. Alex will be around there and hopefully start working on it Monday afternoon :-)

- Alex: JSON Schema bugfix is in, waiting for tools fix to fix the template

#### Tuesday

- Alex: Started working on OptiType modules for DSLv2 conversion
- YARA added, Optitype added to modules

#### Wednesday

- Alex: Added YARA Module for DSLv2 Conversion https://github.com/nf-core/modules/pull/353
- Started template for DSLv2 conversion for hlatyping pipeline
- Still some stuff to do on that one :frowning:

---

# Methylseq - Achievements

#### Monday
- Patrick: added missing modules to nf-core/modules that are required for the dsl2 port
- [bismark/report](https://github.com/nf-core/modules/pull/297)
- [bismark/summary](https://github.com/nf-core/modules/pull/295)
- [bismark/align](https://github.com/nf-core/modules/pull/239)
- Patrick: started to bring [dsl2 draft](https://github.com/nf-core/methylseq/pull/182) into shape
- removed files that are becoming obsolete with dsl2
- Dockerfile, environment.yml, some CI workflows

#### Tuesday
- Patrick: preparing nf-core/methylseq for 1.6 release (hopefully the last dsl1 release)
- template sync 1.13.1 -> https://github.com/nf-core/methylseq/pull/194
- resolved some conda conflicts for newer dependency versions
- resolved a small [issue](https://github.com/nf-core/methylseq/pull/194) that has been on the v1.6 milestone for a while
- PRs ready, but some CI tests seem to be stuck; not sure why

- Patrick: working on dsl2 port
- trying to sync with dsl2 template from @drpatelh's branch of nf-core/tools
- currently getting some cryptic java error with nextflow `21.03.0-edge`; needs more investigation
`Unable to make field private static volatile java.util.List java.nio.file.spi.FileSystemProvider.installedProviders accessible: module java.base does not "opens java.nio.file.spi" to unnamed module @71d44a3` -> nextflow reinstalled solved it

#### Wednesday

- Patrick: opened 1.6 release PR -> https://github.com/nf-core/methylseq/pull/198
- Patrick: worked on dsl2 port aka 2.0dev
- synced with latest dsl2 template
- pushed [dsl2 branch](https://github.com/nf-core/methylseq/tree/dsl2) to nf-core/methylseq repo
- in good shape overall
- workflow summary shows NAs, don't know why
- **please test** :)
---

# Pangenome - Achievements

<!-- State what you achieved here -->
#### Monday
- [x] @subwaystation: Pull request [Important! Template update for nf-core/tools v1.13.1](https://github.com/nf-core/pangenome/pull/51) and issue [Resolve Template Update PR](https://github.com/nf-core/pangenome/issues/54) resolved via pull request https://github.com/nf-core/pangenome/pull/56.
- [x] @subwaystation: Resolved issue [Sync with latest pggb](https://github.com/nf-core/pangenome/issues/52) via pull request https://github.com/nf-core/pangenome/pull/57.
- [x] @subwaystation: Bring `nextflow_schema.json` up to date https://github.com/nf-core/pangenome/pull/58.

#### Tuesday

- [x] @subwaystation: Finished a successful automatic building of a docker image https://github.com/pangenome/pgge/pull/17 required for pgge integration.
- [x] @AndreaGuarracino: add a new flag to compress the output files. https://github.com/nf-core/pangenome/pull/60

#### Wednesday

- [x] @subwaystation: Pull request [Important! Template update for nf-core/tools v1.13.2](https://github.com/nf-core/pangenome/pull/59) resolved via pull request https://github.com/nf-core/pangenome/pull/61.
- [x] @AndreaGuarracino: better graph visualization management https://github.com/nf-core/pangenome/pull/62
- [ ] @subwaystation: Add pgge as a submodule to the pipeline. **FIRST DRAFT READY.**

---

# Ampliseq - Achievements

#### Monday
- Created new branch "DSL2" in nf-core/ampliseq to keep the DSL2 code in development more accessible and [opened PR to dev](https://github.com/nf-core/ampliseq/pull/232) that is solving [transition to DSL2](https://github.com/nf-core/ampliseq/issues/223), [version reporting](https://github.com/nf-core/ampliseq/issues/222), [missing bibliography](https://github.com/nf-core/ampliseq/issues/221), [data for full test](https://github.com/nf-core/ampliseq/issues/215), [update software](https://github.com/nf-core/ampliseq/issues/214), [uses parameter schema](https://github.com/nf-core/ampliseq/issues/205), [classify preprocessed sequences](https://github.com/nf-core/ampliseq/issues/202)
- Pull request [forward preferentially DADA2 classification to downstream analysis](https://github.com/nf-core/ampliseq/pull/233) to open up new possibilities for [taxonomic classification and downstream analysis](https://github.com/nf-core/ampliseq/issues/231)
- Working on flexible and user friendly usage of a mutitude of taxonomic databases.

#### Tuesday
- Added support for more databases, e.g. [gtdb](https://github.com/nf-core/ampliseq/pull/234) and [unite](https://github.com/nf-core/ampliseq/pull/237), through a [module](https://github.com/nf-core/ampliseq/pull/239)
- Fixed [CI tests](https://github.com/nf-core/ampliseq/pull/235)

#### Wednesday
- Fixed downstream analysis, extended options with --input, and fixed linting for nf-core tools 1.13.2 in [one go](https://github.com/nf-core/ampliseq/pull/238)
- Updated the [Unite and GTDB reformatting](https://github.com/nf-core/ampliseq/pull/242)
- Parallellised CI tests for [profiles](https://github.com/nf-core/ampliseq/pull/241)
- Moved various [cutadapt steps into a subworkflow and generate cutadapt summary files](https://github.com/nf-core/ampliseq/pull/246)

---

# Bactmap - Achievements

#### Monday
* @aunderwo prelim version of a pipeline with inputs and a single optional trimming step using fastp working.
* @avantonder new gubbins module
* @avantonder new snpsites module
* @avantonder new iqtree module
* @avantonder new fasttree module
* @alexandregilardet new mash_sketch module

#### Tuesday
next step to contribute other missing modules and stitch them together
* @avantonder new rapidnj module
* @avantonder new raxml-ng module
* @avantonder update CITATIONS.md
* @avantonder update README.md
* @thanhleviet adding a rasusa fastq sampling module
* @aunderwo building modules into pipeline. Pipeline now runs up until the point of having a sorted bam
* @alexandregilardet adding a bcf mpileup/call step

#### Wednesday
* @alexandregilardet added the FastTree step
* @avantonder added gubbins module to nf-core/modules and created PR
* @aunderwo made sure versions were all recorded
* @thanhleviet added genome estimation for rasusa sampling module
* @aunderwo went down a Groovy rabbit hole that he hasn't emerged from!

---

# Bcellmagic - Achievements

#### Monday
* @ggabernet template update to v1.13.1

* @ssnn is working in a temp private repo on some modules for bcellmagic. Template upgrade done. Working now on DSL2 regorganization (modules, params, doc...)

#### Tuesday
* @ggabernet Fixing a couple of issues for DSL2 release

* @ssnn Figured out how to reorganize the code, and use custom container.
---

# Eager - Achievements

#### Monday
- Added requested functionality to pmdtools process (https://github.com/nf-core/eager/issues/349).

#### Tuesday

#### Wednesday

- James merged in 1.13.2 template together with Maxime / Thiseas?
- Fixing MultiVCFAnalyzer not being triggered https://github.com/nf-core/eager/pull/711
---

# mag - Achievements

#### Monday
- [Sabrina] Finished first DSL2 version, just waiting for tests (https://github.com/nf-core/mag/pull/162).

#### Tuesday
- [Sabrina] Finished first DSL2 version for real (https://github.com/nf-core/mag/pull/162, https://github.com/nf-core/mag/pull/171), cleaned params shown in summary, parallellised CI tests (https://github.com/nf-core/mag/issues/170).
- [Sabrina] Compressed output files (https://github.com/nf-core/mag/pull/172)
- [Sabrina] Continued work on gtdb-tk classification (https://github.com/nf-core/mag/issues/173)
- [Sabrina] Start dealing with runtime issue (when computing co-assemblies) of MetaBAT helper script (https://github.com/nf-core/mag/issues/166)

#### Wednesday
- [Sabrina] Fixed bug in splitting unbinned sequences, improved runtime of helper script in MetaBAT2 process (https://github.com/nf-core/mag/pull/175)
- [Sabrina] Continued work on gtdb-tk classification (https://github.com/nf-core/mag/issues/173)

---

# DIAproteomics / MHCquant - Achievements

#### Monday
- [Leon] Template update merge conflicts resolved
- [Leon] Opened two PRs:
- DIAproteomics: https://github.com/nf-core/diaproteomics/pull/103
- MHCquant: https://github.com/nf-core/mhcquant/pull/160

- couple of bugs remain after merging

#### Tuesday
- [Leon] Found an issue in the 1.13.1 template (https://github.com/nf-core/tools/issues/937)
- [Kevin M and Phil E] Solved the issue

- [Leon] Two PRs still open but:
- DIAproteomics and MHCquant tests pass
- bugs were:
- execution rights for scripts in bin folder
- params.name remaining in config

- [Leon] + added an review for:
- Metaboigniter: https://github.com/nf-core/metaboigniter/pull/10

#### Wednesday
- [Leon] Added another review for Metaboigniter
- [Leon] Updated github repository branch protection settings
- [Leon] Adressed reviews by Phil E
- Added more summary docu and references to README
- If tests pass both PRs can be merged tonight

---

# crisprquant - Achievements

#### Monday

- Template updated for input data
- integrated functionality to generate fasta file from library csv file
- Finished including first module

#### Tuesday

- Added indexing, mapping and trimming process
- working on issue with a software's input requirement

#### Wednesday
- fixed issue with input (Thanks Maxime)
- working on first succesful pipeline run

---

# circrna - Achievements

#### Monday
* Template update v13.1
* Add summary log to main.nf
* Stopped pushing directly to head fork - made PRs from local fork :')

#### Tuesday
* Added parameter `--bsj_reads` allowing users to filter by minimum circRNA reads counts.
* Added parameter `--tool_filter` allowing users to take union/intersection of mutliple quantification tools.
* Update help, summary logs, input parameter checks, usage.md, schema.json for above parameters.
* Template update v13.2

#### Wednesday
* reworked `--tool_filter` to take integers instead of strings (i.e output circrnas must be called by at least *n* tools)

@BarryDigby

### Thank you Gisela for leading this workgroup :)

---