# nf-core/blog : pipeline refactoring

---

With the ongoing migration to nf-test, our goal is to enhance the modularity and self-containment of each component in nf-core pipelines, including modules, sub-workflows, and workflows.

**Key Enhancements:**

* **Parallel Testing**: Implementing one test per file to facilitate parallelisation.
* **Strict Tag Provenance**: Ensuring rigorous tag provenance from modules to subworkflows and then to workflows. This approach aids in more targeted and efficient testing when code changes.
* **Dedicated Folders**: Assigning a specific folder for each main script, irrespective of whether they are modules, subworkflows, or workflows.
* **Unified Script Naming**: All main scripts for modules, subworkflows, and workflows will be named main.nf, residing in their respective folders.
* **Tests Alongside Main Scripts**: Placing tests in their own `tests/` folder alongside the main scripts.
* **Distinct emits**: Designating specific emits for modules, subworkflows, and workflows. This practice ensures clear differentiation of output channels for consistent snapshot creation.

**Structure Example:**

Each module, sub-workflow, and workflow script will be isolated in its folder with a dedicated main.nf:

modules/nf-core/tool/subtool/
├── main.nf
├── meta.yml
├── environment.yml
└── tests
    ├── main.nf.test
    └── tags.yml

subworkflows/
├── local
│   └── tool-1_tool-2
│       ├── main.nf
│       ├── main.nf.test.snap
│       └── tags.yml
└── nf-core
    └── tool-1_tool-2_tool-3_tool-4
        ├── main.nf
        ├── meta.yml
        ├── nextflow.config
        └── tests
            ├── main.nf.test
            ├── main.nf.test.snap
            └── tags.yml

workflows/
├── main.nf
├── nextflow.config
└── tests
    ├── main.nf.test
    ├── main.nf.test.snap
    ├── parameter_1.nf.test
    ├── parameter_1.nf.test.snap
    ├── parameter_2.nf.test
    ├── parameter_2.nf.test.snap    
    └── tags.yml

## Shift from `modules.config` to Individual `nextflow.config`:

### For Modules

With the introduction of DSL2, there is a shift in how we manage process configurations for modules. Previously, configurations and tool options for modules were consolidated in a single `modules.config` file. While this approach was functional, it lacked the granularity and isolation that complex pipelines demand.

To address this, we're migrating these configurations into separate `nextflow.config `files for each module. This change has several key benefits:

* **Enhanced Isolation**: By having individual nextflow.config files for each module, we isolate configurations, reducing the risk of conflicts and increasing clarity.
* **Granularity in Configuration**: Separate config files allow for more detailed and module-specific settings, catering to the unique requirements of each module.
* **Easier Maintenance and Updates**: With configurations being module-specific, updating or maintaining a particular module becomes more straightforward, without the need to sift through a central, monolithic configuration file.
* **Improved Flexibility and Reusability**: Modules with their own nextflow.config can be easily shared and reused across different nf-core pipelines, enhancing flexibility and promoting a modular development approach.

### For Subworkflows/Workflows

For integrating module configurations into sub-workflows, individual nextflow.config files from modules are included in the sub-workflow’s nextflow.config:

includeConfig '../../../modules/nf-core/tool-1/subtool/nextflow.config'
includeConfig '../../../modules/nf-core/tool-2/subtool/nextflow.config'
includeConfig '../../../modules/nf-core/tool-3/subtool/nextflow.config'

### Considerations

[add any considerations or potential disadvantages of this approach]

---

## new housekeeping nf-core sub-wfs:

* utils_nextflow_pipeline
* utils_nfcore_pipeline
* utils_nfvalidation_plugin

---

## Changes to github CI

---

## test-data params

`params.modules_test_data_base`

`params.pipelines_test_data_base`

---

conda declarations to have `environment.yml`

---

not including versions in the snapshots -> will have bot for bumping the versions

---