## nf-core/hackathon
Welcome to the pipelines subgroup :rocket:
Thanks to all of you for being here
Special thanks to Adam, Edmund, Harshil, Sateesh
---
We will be working on pipelines: maintaining, updating, refactoring, adding new features.
---
Have a look at our project board:
https://github.com/orgs/nf-core/projects/47/views/4
Join our Slack channel:
https://nfcore.slack.com/channels/hackathon-oct-2023-pipelines
---
Claim a card on the project board to work on,
If there is no card for you, create one.
Don't hesitate to ask us if you have any questions
---
# nf-test
https://code.askimed.com/nf-test/
---
## Why nf-test
- Much simpler than with pytest
- More Nextflow-y and Groovy
- We can easily test at any level
- `functions`
- `modules`
- `workflows`
- `pipeline`I
---
## Modularity
Self contained units (code + config + test)
```text
{modules,subworkflows}/nf-core/<ANY>
├── main.nf
├── meta.yml
└── tests
├── main.nf.test
├── main.nf.test.snap
└── nextflow.config
```
Use the same tests in modules and pipelines!
---
## POC: fetchngs
https://github.com/nf-core/fetchngs/pull/188
---
## Refactoring I
- `conf/modules.config` is split up
- `lib/` has been removed:
- moved to subworkflows
- faster update than TEMPLATE
- native Nextflow code
---
## Refactoring II
- Each main.nf comes with a
- `tests/`
- `tests/main.nf`
- `tests/main.nf.snap` (optional)
- `tests/nextflow.config` (optional)
- Use tag based tests
- `tests/tags.yml` (optional)
- Tests only when relevant changes happen
- Need proper tags
---
## :camera_with_flash: Snapshot testing
https://code.askimed.com/nf-test/docs/assertions/snapshots/
- Baseline set of unit tests to safeguard against regressions caused by changes
- Compare output(s) or file(s) to reference snapshot file(s) stored alongside the tests
---
## How?
1. Create test files:
`nf-test generate <process/workflow/pipeline> main.nf`
2. Fill in the details
* Add `tags`
* provide inputs
* write assertions
3. Create a snapshot
`nf-test test --tag <tag> --profile docker`
5. Update Snapshot (if necessary):
`nf-test test --tag <tag> --update-snapshot`
---
## Issues
---
## Writing tests
When testing for different levels each test need its own file.
(`modules` vs `functions` vs `workflows` vs `pipeline`)
### Ideas for fix:
- write different files:
`main.function.nf.test`
`main.process.nf.test`
`main.workflow.nf.test`
`main.pipeline.nf.test`
---
## Importing tests I
Tests from modules fail in pipeline due to the pipeline config
versions, help and other pipeline specific stuff
### Ideas for fix:
- rewrite the tests to use specific checks (e.g. presence of a string)
- regenerate scripts in pipeline
- add test-specific configuration
---
## Importing tests II
Chaining modules
:point_right: [nf-test `setup` method for chaining](https://code.askimed.com/nf-test/docs/testcases/setup/)
pipeline might not contain the module needed to generate data for the test to run
### Ideas for fix:
- rewrite tests with groovy tar and untar
- rewrite tests so that it won't need the module dependencies
---
## Importing tests III
Missing data
Data in test-datasets/modules, but not in test-datasets/pipeline
### Ideas for fix:
- add data to the test-datasets/pipeline
- rewrite tests so that it won't need the data dependencies
---
## Beginners to nf-test
we recommend starting with generating `process` tests for nf-core/modules (especially the ones used in rnaseq/fetchngs)
Examples: **[nf-core/fastqc](https://github.com/nf-core/modules/tree/master/modules/nf-core/fastqc/tests)**
---
## nf-core maintainers or have used nf-test previously
:point_right: we recommend reviewing [fetchngs](https://github.com/nf-core/fetchngs) / [phageannotator](https://github.com/nf-core/phageannotator)
:point_right: helping with generating `process`/`sub-workflow`/`workflow`/`pipeline` tests for [nf-core/rnaseq](https://github.com/nf-core/rnaseq) or pipeline of your choice
---
## groovy wizard :mage:
- generate `function` tests for nf-core functions
- help finalize the GHA CI
:pray:
---
Thanks again to all of you
<style>
.reveal section img { background:none; border:none; box-shadow:none; }
body {
background-image: url(https://raw.githubusercontent.com/nf-core/logos/master/nf-core-logos/nf-core-logo-square.svg);
background-size: 7.5%;
background-repeat: no-repeat;
background-position: 3% 96%;
background-color: #181a1b;
}
.reveal body {
font-family: 'Roboto', sans-serif;
font-weight: 300;
color: white;
}
.reveal p {
font-family: 'Roboto', sans-serif;
font-weight: 300;
color: white;
}
.reveal h1 {
font-family: 'Roboto', sans-serif;
font-style: bold;
font-weight: 400;
color: #24b064;
font-size: 62px;
}
.reveal h2 {
font-family: 'Roboto', sans-serif;
font-weight: 300;
color: white;
}
.reveal h3 {
font-family: 'Roboto', sans-serif;
font-style: italic;
font-weight: 300;
color: white;
}
.reveal p {
font-family: 'Roboto', sans-serif;
font-weight: 300;
color: white;
}
.reveal li {
font-family: 'Roboto', sans-serif;
font-weight: 300;
color: white;
}
.reveal pre {
background-color: #272822 !important;
display: inline-block;
border-radius: 7px;
color: #aaaba9;
}
.reveal pre code {
color: #eeeeee;
background-color: #272822;
font-size: 100%;
}
.reveal code {
background-color: #272822;
font-size: 75%;
}
.reveal .progress {
color: #24B064;
}
.reveal .controls button {
color: #24B064;
}
.reveal blockquote {
display: block;
position: relative;
width: 90%;
margin: 20px auto;
padding: 5px;
background: rgba(255, 255, 255, 0.05);
box-shadow: 0px 0px 2px rgb(0 0 0 / 20%);
}
.multi-column{
display: grid;
grid-auto-flow: column;
}
</style>
{"title":"nf-core/hackathon/pipelines","description":"Key notion is modularity","contributors":"[{\"id\":\"15ef8394-10ce-4697-80a6-a3eb2914e56c\",\"add\":407,\"del\":81},{\"id\":\"fb193497-1111-470c-a594-827d34b6f673\",\"add\":6155,\"del\":747},{\"id\":\"35a34875-3971-4e5c-ac89-b0a2d4098367\",\"add\":29,\"del\":0},{\"id\":\"f545fc4a-58ec-4fe3-8d2c-4e089141a4dd\",\"add\":1197,\"del\":908}]"}