# Gentle intro to cwltool
## Before we begin
<iframe width="560" height="315" src="https://www.youtube.com/embed/86eY8xs-Vo8" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
Watch the video above for a gentle intro.
## Benefits
Workflows offer numerous advantages, with portability being a key benefit. When designed with flexibility, a workflow can be deployed across various environments, such as locally on a laptop, on a high-performance computing (HPC) cluster like the Brain Image Library or Bridges 2, and in cloud platforms like AWS.
A second significant advantage is the efficiency with which workflows integrate tools, programs, and scripts written in different programming languages, enabling seamless interoperability.
Additionally, workflows can incorporate containers, which can be published to repositories such as DockerHub. This further enhances their portability and flexibility, making workflows adaptable across diverse computational infrastructures.
While workflows provide a robust method for running pipelines on Brain Image Library hardware, users also have the option to utilize traditional methods such as bash or Python scripts to execute their workflows.
:::
```
interact -p RM-shared -n 16 --mem=32000M --time "4:00:00"
```
## Introduction
### Installing cwltool
```
module load anaconda3/2022.10
conda create -n cwltool-tutorial python=3.10
conda activate cwltool-tutorial
pip install --user cwltool cwlref-runner
```
```
(cwltool-tutorial) ➜ ~ which cwl-runner
~/.local/bin/cwl-runner
```
[`cowsay`](https://en.wikipedia.org/wiki/Cowsay) is a tool that pretty prints a cow.
## Example 1: `cowsay`
<img src="https://upload.wikimedia.org/wikipedia/commons/8/80/Cowsay_Typical_Output.png" /><br>
Traditionally we would install the tool locally either using a repository or pip.
For example,
```
pip install cowsay
```
will install the binary in our system.
Since the only input to `cowsay` is a string, a basic CWL workflow document looks like this
```
#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: CommandLineTool
baseCommand: cowsay
inputs:
message:
type: string
inputBinding:
position: 1
outputs: []
```
CWL documents are written either in YAML or JSON. For example, we can create the file `message.cwl`
```
message: Hello world!
```
and use it as input for the workflow
```
cwltool cowsay.cwl message.cwl
INFO /bil/packages/anaconda3/4.11.0/bin/cwltool 3.1.20220210171524
INFO Resolved 'cowsay.cwl' to 'file:///bil/users/icaoberg/code/singularity-cowsay/3.04/cowsay.cwl'
INFO [job cowsay.cwl] /tmp/l7knmpt3$ cowsay \
'Hello world!'
____________
| Hello world! |
============
\
\
^__^
(oo)\_______
(__)\ )\/\
||----w |
|| ||
INFO [job cowsay.cwl] completed success
{}
INFO Final process status is success
```
## `cowsay` on Docker
:::warning
This step cannot run on Brain Image Library hardware since we do not support Docker.
:::
Consider the following `Dockerfile`
```
FROM ubuntu:latest
RUN apt-get update && apt-get install -y cowsay --no-install-recommends && rm -rf /var/lib/apt/lists/*
ENV PATH $PATH:/usr/games
CMD ["cowsay"]
```
The file above creates a container with the `cowsay` binary. I can be built it using the command
```
docker build -t icaoberg/cowsay .
```
and pushed to DockerHub using the command
```
docker push icaoberg/cowsay
```
This is a dummy example but technically now there exists a container with my tool. Now, I can recycle the CWL workflow from before and it to get the container from the repo by adding the lines
```
hints:
DockerRequirement:
dockerPull: icaoberg/cowsay
```
The document now looks like
```
#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: CommandLineTool
requirements:
SubworkflowFeatureRequirement: {}
hints:
DockerRequirement:
dockerPull: icaoberg/cowsay
baseCommand: cowsay
inputs:
message:
type: string
inputBinding:
position: 1
outputs: []
```
and running it will produce the same results as the previous example
```
cwltool cowsay2.cwl message.cwl
INFO /Users/icaoberg/opt/anaconda3/bin/cwltool 3.1.20220210171524
INFO Resolved 'cowsay2.cwl' to 'file:///Users/icaoberg/Documents/code/singularity-cowsay/3.04/cowsay2.cwl'
INFO [job cowsay2.cwl] /private/tmp/docker_tmpr_rjhbrj$ docker \
run \
-i \
--mount=type=bind,source=/private/tmp/docker_tmpr_rjhbrj,target=/xJHVRn \
--mount=type=bind,source=/private/tmp/docker_tmpzpu0ulbd,target=/tmp \
--workdir=/xJHVRn \
--read-only=true \
--user=501:20 \
--rm \
--cidfile=/private/tmp/docker_tmp07wk4ale/20220309145946-550721.cid \
--env=TMPDIR=/tmp \
--env=HOME=/xJHVRn \
icaoberg/cowsay \
cowsay \
'Hello world!'
______________
< Hello world! >
--------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
INFO [job cowsay2.cwl] Max memory used: 0MiB
INFO [job cowsay2.cwl] completed success
{}
INFO Final process status is success
```
:::info
Even though we do not support Docker, you can try installing [`uDocker`](https://github.com/indigo-dc/udocker).
:::
## `cowsay` on Singularity
The main issue is that most HPC clusters do not support Docker and prefer Singularity or Apptainers. However, if the Docker image in DockerHub has proper entrypoints, then you could simply use the `--singularity` option to ask CWL tools to convert the Docker image to Singularity.
:::warning
If the Docker image does not a proper entry point this step might fail if you are not aware of how the image was built.
Only use vetted images or public images whose Dockerfile you have seen and trust.
:::
Using the option
```
cwltool --singularity cowsay2.cwl message.cwl
```
will run the workflow
```
cwltool --singularity cowsay2.cwl message.cwl
INFO /bil/packages/anaconda3/4.11.0/bin/cwltool 3.1.20220210171524
INFO Resolved 'cowsay2.cwl' to 'file:///bil/users/icaoberg/code/singularity-cowsay/3.04/cowsay2.cwl'
INFO ['singularity', 'pull', '--force', '--name', 'icaoberg_cowsay.sif', 'docker://icaoberg/cowsay']
INFO: Converting OCI blobs to SIF format
INFO: Starting build...
Getting image source signatures
Copying blob 7c3b88808835 done
Copying blob 6b7a6ea66907 done
Copying config 063d227371 done
Writing manifest to image destination
Storing signatures
2022/03/09 15:17:27 info unpack layer: sha256:7c3b88808835aa80f1ef7f03083c5ae781d0f44e644537cd72de4ce6c5e62e00
2022/03/09 15:17:28 info unpack layer: sha256:6b7a6ea669076a74f122534da10e4e459f36777854e8e1529564d31c685fd9ea
INFO: Creating SIF file...
INFO [job cowsay2.cwl] /tmp/ceztlkix$ singularity \
--quiet \
exec \
--contain \
--ipc \
--cleanenv \
--pid \
--home \
/tmp/ceztlkix:/qxOGcV \
--bind \
/tmp/o9hfm5tx:/tmp \
--pwd \
/qxOGcV \
/bil/users/icaoberg/code/singularity-cowsay/3.04/icaoberg_cowsay.sif \
cowsay \
'Hello world!'
______________
< Hello world! >
--------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
INFO [job cowsay2.cwl] completed success
{}
INFO Final process status is success
```
but will create a Singularity image file on disk.
### Adding more options
`cowsay` has more options than just the input string.
```
cowsay(6) Games Manual cowsay(6)
NAME
cowsay/cowthink - configurable speaking/thinking cow (and a bit more)
SYNOPSIS
cowsay [-e eye_string] [-f cowfile] [-h] [-l] [-n] [-T tongue_string] [-W column] [-bdg‐
pstwy]
```
A `cowfile` is used to change the picture. For example, running the command
```
➜ code cowsay -f flaming-sheep "Hello World\!"
______________
< Hello World! >
--------------
\ . . .
\ . . . ` ,
\ .; . : .' : : : .
\ i..`: i` i.i.,i i .
\ `,--.|i |i|ii|ii|i:
UooU\.'@@@@@@`.||'
\__/(@@@@@@@@@@)'
(@@@@@@@@)
`YY~~~~YY'
|| ||
```
will print a flaming sheep.
In this example, we will expose the `[-f cowfile]` argument by adding the lines
```
format:
type: string
inputBinding:
position: 1
prefix: -f
default: "flaming-sheep"
```
to the input block, making the workflow look like
```
#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: CommandLineTool
requirements:
SubworkflowFeatureRequirement: {}
hints:
DockerRequirement:
dockerPull: icaoberg/cowsay
baseCommand: "cowsay"
inputs:
message:
type: string
inputBinding:
position: 2
format:
type: string
inputBinding:
position: 1
prefix: -f
default: "flaming-sheep"
outputs: []
```
Then you can run it
```
05:03:16 icaoberg@workshop 3.04 ±|master ✗|→ cwltool --singularity cowsay3.cwl message3.cwl
INFO /bil/users/icaoberg/.local/bin/cwltool 3.1.20220224085855
INFO Resolved 'cowsay3.cwl' to 'file:///bil/users/icaoberg/code/singularity-cowsay/3.04/cowsay3.cwl'
INFO Using local copy of Singularity image found in /bil/users/icaoberg/code/singularity-cowsay/3.04
INFO [job cowsay3.cwl] /tmp/g2zknke0$ singularity \
--quiet \
exec \
--contain \
--ipc \
--cleanenv \
--pid \
--home \
/tmp/g2zknke0:/lzqerl \
--bind \
/tmp/fkb0phq4:/tmp \
--pwd \
/lzqerl \
/bil/users/icaoberg/code/singularity-cowsay/3.04/icaoberg_cowsay.sif \
cowsay \
-f \
flaming-sheep \
'Hello world!'
______________
< Hello world! >
--------------
\ . . .
\ . . . ` ,
\ .; . : .' : : : .
\ i..`: i` i.i.,i i .
\ `,--.|i |i|ii|ii|i:
UooU\.'@@@@@@`.||'
\__/(@@@@@@@@@@)'
(@@@@@@@@)
`YY~~~~YY'
|| ||
INFO [job cowsay3.cwl] completed success
{}
INFO Final process status is success
```
Keep in mind your input argument `message3.yml` now looks like this
```
message: Hello world!
format: flaming-sheep
```
You can choose to expose as many input arguments as you want or set default values.
:::warning
Anything below this line I am still working on
:::
## Mixing and matching
Consider the following workflow, `fortune.cwl`
```
#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: CommandLineTool
requirements:
SubworkflowFeatureRequirement: {}
hints:
DockerRequirement:
dockerPull: grycap/cowsay
baseCommand: /usr/games/fortune
inputs: []
outputs: []
```
which does something like
```
cwltool --singularity fortune.cwl
INFO /bil/users/icaoberg/.local/bin/cwltool 3.1.20220224085855
INFO Resolved 'fortune.cwl' to 'file:///bil/users/icaoberg/code/singularity-cowsay/3.04/fortune.cwl'
INFO Using local copy of Singularity image found in /bil/users/icaoberg/code/singularity-cowsay/3.04
INFO [job fortune.cwl] /tmp/lppo005u$ singularity \
--quiet \
exec \
--contain \
--ipc \
--cleanenv \
--pid \
--home \
/tmp/lppo005u:/jzrcyZ \
--bind \
/tmp/l0squr2a:/tmp \
--pwd \
/jzrcyZ \
/bil/users/icaoberg/code/singularity-cowsay/3.04/grycap_cowsay.sif \
/usr/games/fortune
Q: How many lawyers does it take to change a light bulb?
A: You won't find a lawyer who can change a light bulb. Now, if
you're looking for a lawyer to screw a light bulb...
INFO [job fortune.cwl] completed success
{}
INFO Final process status is success
```
Sign in with Wallet
Connect another wallet