---
# System prepended metadata

title: subworkflows planning
tags: [subworkflows, ' meeting notes', ' technical']

---

---
title: subworkflows planning
tags: subworkflows, meeting notes, technical
---



# Subworkflows planning

### 2020-07-05

> _Present:_ @ewels, Mahesh, @mashehu, @maxulysse, Friederike, Júlia, @emiller, Erik, @joseespinosa, Gisela, Harshil, Matthias H., Matthias De Smet, Alex, James, 


## Summary

1. Plan is to establish subworkflow tooling before the hackathon so the focus can then be to add subworkflows
2. Julia & Eric are currently working on tools codebase
3. Need to figure out how to do this
    - Build a PoC first, `bamsortsamtools`
    - Work on this and get as much agreement as possible about file structure
    - Roll updates for reminaing subworkflows
    - Update tools

## TODO

- [ ] Fix CI testing for subworkflows [Historical PR 662](https://github.com/nf-core/modules/pull/662)
    - **@emiller** to mess around with anchors to try to get an example to work or something? Revisit and see what breaks.
- [ ] **Harshil:** [Update all subworkflows to new NF syntax](https://github.com/nf-core/modules/pull/1187)
- [ ] Try out PoC on **nf-core/testpipeline**
- [ ] **Maxime:** Add more Sarek subworkflows into nf-core/modules
- [ ] **Harshil** to ping on Slack `#subworkflows` when ready for review

## Code structure

- Will we have a new shared repo? **nf-core/subworkflows**?
    - Can't have modules + subworkflows in the same PR then
    - Website: Have subworkflows tab
    - **Decision:** No, stick with nf-core/modules
- Relative paths to modules? OR reinstall / bundle within the subworkflow directory in the repo
    - **1. Relative paths to modules**:
        - Already have a proof of concept with this structure
        - **pros**:
            - more simple structure
            - just one module version x pipeline
            - CI: Force all subworkflows to be updated when a module is updated
        - **cons**:
            - Forces all subworkflows in a pipeline to use the same version of a module
            - when one module gets updated, potentially breaks all subworkflows
            - need to first update module with PR, then use commit ID within subworkflow?
    - **2. Bundle within subworkflow**
        - **pros**:
            - allows for different module versions in different subworkflows
            - allows for nested subworkflows
        - **cons**:
            - ~~issue with testing?~~
            - potential risk of having different tool versions / module versions within pipeline, that could be solved with linting.
    - Let's try to avoid mixing local and shared modules / subworkflows
    - **Decision**: Start off with option 1 and see how things go.

## CI testing

- Anchors + pytest-workflow?
    - Edmund tried this, but Harshil had to strip it out as it was breaking stuff.
    - Would like to mimic how CI works for modules

## General

- Computational efficiency: Batching groups of processes, like Snakemake
    - https://github.com/nf-core/modules/issues/1179 (should a subworkflow be implemented as a process)
    - Coming at some point in core Nextflow, though not very high priority, so may take a while.

