---
title: subworkflows planning
tags: subworkflows, meeting notes, technical
---

# Subworkflows planning

### 2020-07-05

> _Present:_ @ewels, Mahesh, @mashehu, @maxulysse, Friederike, Júlia, @emiller, Erik, @joseespinosa, Gisela, Harshil, Matthias H., Matthias De Smet, Alex, James,

## Summary

1. Plan is to establish subworkflow tooling before the hackathon so the focus can then be to add subworkflows
2. Julia & Eric are currently working on tools codebase
3. Need to figure out how to do this
- Build a PoC first, `bamsortsamtools`
- Work on this and get as much agreement as possible about file structure
- Roll updates for reminaing subworkflows
- Update tools

## TODO

- [ ] Fix CI testing for subworkflows [Historical PR 662](https://github.com/nf-core/modules/pull/662)
- **@emiller** to mess around with anchors to try to get an example to work or something? Revisit and see what breaks.
- [ ] **Harshil:** [Update all subworkflows to new NF syntax](https://github.com/nf-core/modules/pull/1187)
- [ ] Try out PoC on **nf-core/testpipeline**
- [ ] **Maxime:** Add more Sarek subworkflows into nf-core/modules
- [ ] **Harshil** to ping on Slack `#subworkflows` when ready for review

## Code structure

- Will we have a new shared repo? **nf-core/subworkflows**?
- Can't have modules + subworkflows in the same PR then
- Website: Have subworkflows tab
- **Decision:** No, stick with nf-core/modules
- Relative paths to modules? OR reinstall / bundle within the subworkflow directory in the repo
- **1. Relative paths to modules**:
- Already have a proof of concept with this structure
- **pros**:
- more simple structure
- just one module version x pipeline
- CI: Force all subworkflows to be updated when a module is updated
- **cons**:
- Forces all subworkflows in a pipeline to use the same version of a module
- when one module gets updated, potentially breaks all subworkflows
- need to first update module with PR, then use commit ID within subworkflow?
- **2. Bundle within subworkflow**
- **pros**:
- allows for different module versions in different subworkflows
- allows for nested subworkflows
- **cons**:
- ~~issue with testing?~~
- potential risk of having different tool versions / module versions within pipeline, that could be solved with linting.
- Let's try to avoid mixing local and shared modules / subworkflows
- **Decision**: Start off with option 1 and see how things go.

## CI testing

- Anchors + pytest-workflow?
- Edmund tried this, but Harshil had to strip it out as it was breaking stuff.
- Would like to mimic how CI works for modules

## General

- Computational efficiency: Batching groups of processes, like Snakemake
- https://github.com/nf-core/modules/issues/1179 (should a subworkflow be implemented as a process)
- Coming at some point in core Nextflow, though not very high priority, so may take a while.