Who is it for?

If you work in a scenario where you need to process hundred’s of DNA/RNA sequenced samples every week then nextflow is for you!

Many bioinformatics tasks or processes that you run, for example running fastqc are sequential by default and will eat away a lot of compute time as you wait to run the next step in your workflow. e.g. to run fastqc on all of your fastq files, you may use a for loop like below to get the results.

for i in *.fastq.gz ; do fastqc $i ; done

Note that this loop is sequential, meaning it will only process one file at a time and if fastqc takes 10 minutes to process one fastq file, it can add up very quickly as you process hundreds of samples altogether.

Why use nextflow?

Many bioinformatics pipelines comprise of various tools that one uses in a dataflow programming manner to get the final results. Some of these tools may be accessible with the BASH interpreter, some are written in python and will require the use of a python interpreter and some are written in R and will require R console for data analysis. Often times many bioinformaticians or data analysts have custom scripts that they also run on their samples in order to get desired results.

Doing this requires jumping from one programming interpreter to another which is not possible to do from a single script file, however nextflow makes this easier and allows the user to define separate code blocks in its processes and makes it super-efficient to chain all of your analysis steps into one main file, which again is a huge benefit.

Pipelines

In practice, a Nextflow pipeline is made by joining together different processes. Each process can be written in any scripting language that can be executed by the Linux platform (Bash, Perl, Ruby, Python, etc.).

Processes are executed independently and are isolated from each other. Any process can define one or more channels as an input and output. The interaction between these processes, and ultimately the pipeline execution flow itself, is implicitly defined by these input and output declarations.

Let's take a look at a nextflow script

1. The code begins with a shebang, which declares Nextflow as the interpreter.
2. Declares a parameter greeting that is initialized with the value 'Hello world!'.
3. Initializes a channel labelled greeting_ch, which contains the value from params.greeting. Channels are the input type for processes in Nextflow.
4. Begins the first process, defined as SPLITLETTERS.
5. Input declaration for the SPLITLETTERS process. Inputs can be values (val), files or paths (path), or other qualifiers (see here). 6 Tells the process to expect an input value (val), that we assign to the variable 'x'.
6. Output declaration for the SPLITLETTERS process.
7. Tells the process to expect an output file(s) (path), with a filename starting with 'chunk_*', as output from the script. The process sends the output as a channel.
8. Three double quotes initiate the code block to execute in this process.
9. Code to execute — printing the input value x (called using the dollar symbol [$] prefix), splitting the string into chunks with a length of 6 characters ("Hello " and "world!"), and saving each to a file (chunk_aa and chunk_ab).
10. Three double quotes end the code block.
11. End of the first process block.
12. Begins the second process, defined as CONVERTTOUPPER.
13. Input declaration for the CONVERTTOUPPER process.
14. Tells the process to expect an input file(s) (path; i.e. chunk_aa and chunk_ab), that we assign to the variable 'y'.
15. Output declaration for the CONVERTTOUPPER process.
16. Tells the process to expect output as standard output (stdout) and send this output as a channel.
17. Three double quotes initiate the code block to execute in this process.
18. Script to read files (cat) using the '$y' input variable, then pipe to uppercase conversion, outputting to standard output.
19. Three double quotes end the code block.
20. End of first process block.
21. Start of the workflow scope, where each process can be called.
22. Execute the process SPLITLETTERS on the greeting_ch (aka greeting channel), and store the output in the channel letters_ch.
23. Execute the process CONVERTTOUPPER on the letters channel letters_ch, which is flattened using the operator .flatten(). This transforms the input channel in such a way that every item is a separate element. We store the output in the channel results_ch.
24. The final output (in the results_ch channel) is printed to screen using the view operator (appended onto the channel name).
25. End of the workflow scope.

In practice

Now copy the above example into your favourite text editor and save it to a file named hello.nf.

Execute the script by entering the following command in your terminal:

nextflow run hello.nf

The output will look similar to the text shown below:

Launching `hello.nf` [fabulous_torricelli] DSL2 - revision: 197a0e289a
executor >  local (3)
[c8/c36893] process > SPLITLETTERS (1)   [100%] 1 of 1 ✔
[1a/3c54ed] process > CONVERTTOUPPER (2) [100%] 2 of 2 ✔
HELLO
WORLD!

• 1: The Nextflow version executed.
• 2: The script and version names.
• 3: The executor used (in the above case: local).
• 4: The first process is executed once (1). The line starts with a unique hexadecimal value (see TIP below), and ends with the percentage and job completion information.
• 5: The second process is executed twice (2) (once for chunk_aa, once for chunk_ab).
• 6-7: The result string from stdout is printed.

Further study

There is a good tutorial for making a RNA-seq analysis pipeline at https://training.seqera.io/#_simple_rna_seq_pipeline. Try to follow it.