---
# System prepended metadata

title: Docker and Singularity course

---

# Docker and Singularity course

Write questions, notes etc.


## References
* [Dockerfile reference](https://docs.docker.com/engine/reference/builder/)
* [`docker logs`](https://docs.docker.com/config/containers/logging/)
* [E-learning material for consultance](https://elearning.bits.vib.be/courses/introduction-to-docker/lessons/introduction-to-docker/)

## Notes

* Run a command in a docker container as your own user and group
  `docker run --user $(id -u):$(id -g)`
 
* If the `~/.singularity` directory does not exist you can `singularity cache clean` to create it
* Singularity configuration file
  `/usr/local/etc/singularity/singularity.conf`
  There is also an environment variable to configure [bind mounting](https://docs.sylabs.io/guides/3.0/user-guide/bind_paths_and_mounts.html): `$SINGULARITY_BIND`

# Docker introduction (1st day)

## Solutions for Exercise 2:

`docker run --rm -it --name daria_docker -v $PWD/data:/data quay.io/biocontainers/fastqc:0.11.9--0 fastqc /data/ecoli_1.fastq.gz`
> Well done! [name=MaybeJustJames]

## Solutions for Exercise 3

```bash
docker run --rm -it --name daria_docker -v $PWD/data:/scratch quay.io/biocontainers/fastqc:0.11.9--0 /bin/bash

echo "file1" >> /scratch/file1.txt # (root root)
echo "file2" >> data/file2.txt # (training training) on host;

$ ls -lh
-rw-rw-r--    1 default  default        6 Oct 24 11:51 /scratch/file2.txt # on container
```

```bash
docker run --rm -it -u 1000 --name fastqc --volume $PWD/data:/scratch quay.io/biocontainers/fastqc:0.11.9--0 /bin/bash

touch file3.txt
# permissions on host will have user instead of root
```

The `$USER` inside the docker container shares a `uid` but __not__ a user name.
```bash
# Does not work
docker run -u training --rm -it --name daria_docker -v $PWD:/data quay.io/biocontainers/fastqc:0.11.9--0 touch /data/file3.txt
```
```bash
# Works!
docker run -u default --rm -it --name daria_docker -v $PWD:/data quay.io/biocontainers/fastqc:0.11.9--0 touch /data/file3.txt

docker run --user $(id -u):$(id -g)
```


## Solutions to Exercise 4

```bash
#Exercise 4
#Set new WD
docker run --name ex4 -w /scratch/ quay.io/biocontainers/fastqc:0.11.9--0 touch file4.txt
# Run in both files at once
docker run --name ex4 -w /scratch/ -u $(id -u):$(id -g) --volume $(pwd)/data/:/data/ quay.io/biocontainers/fastqc:0.11.9--0 bash -c "fastqc /data/ecoli_*.fastq.gz"
#Get working dir
docker inspect ex4 | grep Working

#Pull trimmomatic
docker pull quay.io/biocontainers/trimmomatic:0.35--6
#Now run trimmoatic
docker run --rm --name ex4_1 -w /scratch/ -u $(id -u):$(id -g) --volume $(pwd)/data/:/data/ quay.io/biocontainers/trimmomatic:0.35--6 sh -c "trimmomatic SE -threads 1 -phred33  /data/ecoli_1.fastq.gz /data/ecoli_1_output.fastq ILLUMINACLIP:$ADAPTERS:2:30:10 SLIDINGWINDOW:4:5 LEADING:5 TRAILING:5 MINLEN:25"
```

## ...

```dockerfile=
RUN pip install pydantic
```

link last exercise day 1:https://github.com/vibbits/containers-workshop/edit/main/docker/Dockerfile

# Singulary introduction (2nd day)
If the `~/.singularity` directory does not exist to creat it you can do:
`singularity cache clean`

* Pull `fastqc`
```singularity pull --name fastqc-0.11.9--0.sif https://depot.galaxyproject.org/singularity/fastqc:0.11.9--0```

* Config file: info about mount volume also findable there
`/usr/local/etc/singularity/singularity.conf`

* environment variable SINGULARITY_BIND or SINGULARITY_BINDPATH

* From https://docs.sylabs.io/guides/3.0/user-guide/bind_paths_and_mounts.html



Use this image for exercise 0-2
singularity pull hello-world.sif shub://vsoch/hello-world

### VI editor cheat Sheet
> https://www.atmos.albany.edu/daes/atmclasses/atm350/vi_cheat_sheet.pdf

### Exercise 0
What do you notice about the ownership of files in a container started from the hello-world image? (e.g. take a look at the ownership of files in the root directory (/))

> #### Answer 0.1 : 
owned by root

```
Singularity> cd /
Singularity> ls -lah
total 45K
drwxr-xr-x   1 training training   60 Oct 26 09:32 .
drwxr-xr-x   1 training training   60 Oct 26 09:32 ..
-rw-r--r--   1 root     root        0 Jul 19  2019 .coredone
lrwxrwxrwx   1 root     root       27 Jul 17  2019 .exec -> .singularity.d/actions/exec
lrwxrwxrwx   1 root     root       26 Jul 17  2019 .run -> .singularity.d/actions/run
lrwxrwxrwx   1 root     root       28 Jul 17  2019 .shell -> .singularity.d/actions/shell
drwxr-xr-x   5 root     root      127 Jul 19  2019 .singularity.d
lrwxrwxrwx   1 root     root       27 Jul 17  2019 .test -> .singularity.d/actions/test
drwxr-xr-x   2 root     root     1.7K May 15  2019 bin
drwxr-xr-x   2 root     root        3 Apr 10  2014 boot
drwxr-xr-x  15 root     root     4.1K Oct 26 07:02 dev
lrwxrwxrwx   1 root     root       36 Jul 17  2019 environment -> .singularity.d/env/90-environment.sh
drwxr-xr-x  61 root     root     2.1K Jul 19  2019 etc
drwxr-xr-x   1 training training   60 Oct 26 09:32 home
drwxr-xr-x  12 root     root      241 May 15  2019 lib
drwxr-xr-x   2 root     root       43 May 15  2019 lib64
drwxr-xr-x   2 root     root        3 May 15  2019 media
drwxr-xr-x   2 root     root        3 Apr 10  2014 mnt
drwxr-xr-x   2 root     root        3 May 15  2019 opt
dr-xr-xr-x 226 root     root        0 Oct 26 07:01 proc
-rwxr-xr-x   1 root     root       46 Jul 19  2019 rawr.sh
drwx------   2 root     root       46 Jul 19  2019 root
drwxr-xr-x   8 root     root      141 Jul 19  2019 run
drwxr-xr-x   2 root     root     2.1K Jul 19  2019 sbin
lrwxrwxrwx   1 root     root       24 Jul 17  2019 singularity -> .singularity.d/runscript
drwxr-xr-x   2 root     root        3 May 15  2019 srv
dr-xr-xr-x  12 root     root        0 Oct 26 07:01 sys
drwxrwxrwt   9 root     root      40K Oct 26 09:33 tmp
drwxr-xr-x  10 root     root      150 May 15  2019 usr
drwxr-xr-x  11 root     root      160 May 15  2019 var
```
> #### Answer 0.2: 
```
drwxr-xr-x   2 nobody  nogroup    0 May 15  2019 bin
drwxr-xr-x   2 nobody  nogroup    0 Apr 11  2014 boot
drwxr-xr-x  21 nobody  nogroup 4080 Oct 26 09:25 dev
lrwxrwxrwx   1 stefaan stefaan   36 Oct 26 11:39 environment -> .singularity.d/env/90-environment.sh
drwxr-xr-x  61 nobody  nogroup    0 Jul 19  2019 etc
drwxr-xr-x   3 stefaan stefaan   60 Oct 26 11:39 home
drwxr-xr-x  12 nobody  nogroup    0 May 15  2019 lib
drwxr-xr-x   2 nobody  nogroup    0 May 15  2019 lib64
drwxr-xr-x   2 nobody  nogroup    0 May 15  2019 media
drwxr-xr-x   2 nobody  nogroup    0 Apr 11  2014 mnt
drwxr-xr-x   2 nobody  nogroup    0 May 15  2019 opt
dr-xr-xr-x 390 nobody  nogroup    0 Oct 26 09:25 proc
-rwxr-xr-x   1 nobody  nogroup   46 Jul 19  2019 rawr.sh
drwx------   2 nobody  nogroup    0 Jul 19  2019 root
drwxr-xr-x   8 nobody  nogroup    0 Jul 19  2019 run
drwxr-xr-x   2 nobody  nogroup    0 Jul 19  2019 sbin
lrwxrwxrwx   1 stefaan stefaan   24 Oct 26 11:39 singularity -> .singularity.d/runscript
drwxr-xr-x   2 nobody  nogroup    0 May 15  2019 srv
dr-xr-xr-x  13 nobody  nogroup    0 Oct 26 11:28 sys
drwxrwxrwt  20 nobody  nogroup  420 Oct 26 11:36 tmp
drwxr-xr-x  10 nobody  nogroup    0 May 15  2019 usr
drwxr-xr-x  11 nobody  nogroup    0 May 15  2019 var
```

### Exercise 1 
In this container, try editing (for example using the editor vi which should be available in the container) the /rawr.sh file. What do you notice?

If you’re not familiar with vi there are many quick reference pages online showing the main commands for using the editor, for example this one.

> #### Answer:

File was read-only; editing was not possible

### Exercise 2 
In your home directory within the container shell, try and create a simple text file. Is it possible to do this? If so, why? If not, why not?! If you can successfully create a file, what happens to it when you exit the shell and the container shuts down?

> #### Answer:
you could create a file which is owned by training user. When you exit the singularity shell, it is still there.

From host:
```
$ ls -lah ~/singularity
total 313M
drwxrwxr-x  2 training training 4.0K Oct 26 08:48 .
drwxr-xr-x 17 training training 4.0K Oct 26 09:33 ..
-rwxrwxr-x  1 training training 253M Oct 26 08:37 fastqc-0.11.9--0.sif
-rw-rw-r--  1 training training    0 Oct 26 08:48 file1.txt
-rwxrwxr-x  1 training training  60M Oct 26 08:38 hello-world.sif
```

### Exercise 3
Try to run the fastqc command by using the fastqc singularity image from the Galaxy depot on both example fastq files from day 1.

> #### Answer (also works for HPC; small files): 
`singularity exec fastqc-0.11.9--0.sif fastqc ~/data/ecoli_*gz`


> #### Daria: doesn't work
`singularity exec -B ~/data:/scratch fastqc-0.11.9--0.sif ls ./scratch/`

> #### Daria: works
`singularity exec -B ~/data:/scratch fastqc-0.11.9--0.sif ls /scratch/`

> #### Teresa: Answer does not work
`singularity exec -B ~/datatest:/scratch sif-images/fastqc-0.11.9--0.sif fastqc /scratch/*.gz`

glob pattern * is evaluated before running singularity, so scratch is not mounted yet

> #### works:

`singularity exec -B ~/datatest:/scratch sif-images/fastqc-0.11.9–0.sif bash -c "fastqc /scratch/*.gz`

`singularity exec -B ~/datatest:/scratch sif-images/fastqc-0.11.9–0.sif fastqc /scratch/ecoli_1.gz`

> #### Answer with -B:

```
$ singularity exec -B ~/containers/data/:/scratch ~/singularity/fastqc-0.11.9--0.sif fastqc /scratch/ecoli_1.fastq.gz
WARNING: Skipping mount /usr/local/var/singularity/mnt/session/etc/resolv.conf [files]: /etc/resolv.conf doesn't exist in container
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LANGUAGE = (unset),
        LC_ALL = (unset),
        LANG = "C.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
Started analysis of ecoli_1.fastq.gz
Approx 5% complete for ecoli_1.fastq.gz
Approx 10% complete for ecoli_1.fastq.gz
Approx 15% complete for ecoli_1.fastq.gz
Approx 20% complete for ecoli_1.fastq.gz
Approx 25% complete for ecoli_1.fastq.gz
Approx 30% complete for ecoli_1.fastq.gz
Approx 35% complete for ecoli_1.fastq.gz
Approx 40% complete for ecoli_1.fastq.gz
Approx 45% complete for ecoli_1.fastq.gz
Approx 50% complete for ecoli_1.fastq.gz
Approx 55% complete for ecoli_1.fastq.gz
Approx 60% complete for ecoli_1.fastq.gz
Approx 65% complete for ecoli_1.fastq.gz
Approx 70% complete for ecoli_1.fastq.gz
Approx 75% complete for ecoli_1.fastq.gz
Approx 80% complete for ecoli_1.fastq.gz
Approx 85% complete for ecoli_1.fastq.gz
Approx 90% complete for ecoli_1.fastq.gz
Approx 95% complete for ecoli_1.fastq.gz
Approx 100% complete for ecoli_1.fastq.gz
Analysis complete for ecoli_1.fastq.gz
```

From host:

```
$ ls -lah  ~/containers/data/
total 7.5M
drwxrwxr-x 2 training training 4.0K Oct 26 10:03 .
drwxrwxr-x 5 training training 4.0K Oct 24 13:50 ..
-rw-r--r-- 1 training training 3.3M Jan 29  2021 ecoli_1.fastq.gz
-rw-rw-r-- 1 training training 600K Oct 26 10:03 ecoli_1_fastqc.html
-rw-rw-r-- 1 training training 344K Oct 26 10:03 ecoli_1_fastqc.zip
-rw-r--r-- 1 training training 3.3M Jan 29  2021 ecoli_2.fastq.gz
```




#### Activity - To be pulled (pay attetion in the folder you will pull it)

`singularity pull python-3.9.6.sif docker://python:3.9.6-slim-buster`

#### Using it remotly

singularity exec docker://ncbi/blast:2.10.1 blastp -version

#### Sandbox

```singularity build --sandbox ./sandbox docker://ubuntu:18.04```

```singularity exec ./sandbox bash -c 'apt-get update && apt-get install python'```

https://elearning.bits.vib.be/courses/introduction-to-docker/lessons/building-singularity-images/topic/using-singularity-recipes/

![](https://i.imgur.com/bghPvZx.png)

![](https://i.imgur.com/K7Y86Xv.png)


#### Syntax highlighting of Singularity definition in VScode
https://github.com/onnovalkering/vscode-singularity

example with debootstrap
install it in the VM template
sudo apt-get install debootstrap