Docker and Singularity course

# Docker and Singularity course Write questions, notes etc. ## References * [Dockerfile reference](https://docs.docker.com/engine/reference/builder/) * [`docker logs`](https://docs.docker.com/config/containers/logging/) * [E-learning material for consultance](https://elearning.bits.vib.be/courses/introduction-to-docker/lessons/introduction-to-docker/) ## Notes * Run a command in a docker container as your own user and group `docker run --user $(id -u):$(id -g)` * If the `~/.singularity` directory does not exist you can `singularity cache clean` to create it * Singularity configuration file `/usr/local/etc/singularity/singularity.conf` There is also an environment variable to configure [bind mounting](https://docs.sylabs.io/guides/3.0/user-guide/bind_paths_and_mounts.html): `$SINGULARITY_BIND` # Docker introduction (1st day) ## Solutions for Exercise 2: `docker run --rm -it --name daria_docker -v $PWD/data:/data quay.io/biocontainers/fastqc:0.11.9--0 fastqc /data/ecoli_1.fastq.gz` > Well done! [name=MaybeJustJames] ## Solutions for Exercise 3 ```bash docker run --rm -it --name daria_docker -v $PWD/data:/scratch quay.io/biocontainers/fastqc:0.11.9--0 /bin/bash echo "file1" >> /scratch/file1.txt # (root root) echo "file2" >> data/file2.txt # (training training) on host; $ ls -lh -rw-rw-r-- 1 default default 6 Oct 24 11:51 /scratch/file2.txt # on container ``` ```bash docker run --rm -it -u 1000 --name fastqc --volume $PWD/data:/scratch quay.io/biocontainers/fastqc:0.11.9--0 /bin/bash touch file3.txt # permissions on host will have user instead of root ``` The `$USER` inside the docker container shares a `uid` but __not__ a user name. ```bash # Does not work docker run -u training --rm -it --name daria_docker -v $PWD:/data quay.io/biocontainers/fastqc:0.11.9--0 touch /data/file3.txt ``` ```bash # Works! docker run -u default --rm -it --name daria_docker -v $PWD:/data quay.io/biocontainers/fastqc:0.11.9--0 touch /data/file3.txt docker run --user $(id -u):$(id -g) ``` ## Solutions to Exercise 4 ```bash #Exercise 4 #Set new WD docker run --name ex4 -w /scratch/ quay.io/biocontainers/fastqc:0.11.9--0 touch file4.txt # Run in both files at once docker run --name ex4 -w /scratch/ -u $(id -u):$(id -g) --volume $(pwd)/data/:/data/ quay.io/biocontainers/fastqc:0.11.9--0 bash -c "fastqc /data/ecoli_*.fastq.gz" #Get working dir docker inspect ex4 | grep Working #Pull trimmomatic docker pull quay.io/biocontainers/trimmomatic:0.35--6 #Now run trimmoatic docker run --rm --name ex4_1 -w /scratch/ -u $(id -u):$(id -g) --volume $(pwd)/data/:/data/ quay.io/biocontainers/trimmomatic:0.35--6 sh -c "trimmomatic SE -threads 1 -phred33 /data/ecoli_1.fastq.gz /data/ecoli_1_output.fastq ILLUMINACLIP:$ADAPTERS:2:30:10 SLIDINGWINDOW:4:5 LEADING:5 TRAILING:5 MINLEN:25" ``` ## ... ```dockerfile= RUN pip install pydantic ``` link last exercise day 1:https://github.com/vibbits/containers-workshop/edit/main/docker/Dockerfile # Singulary introduction (2nd day) If the `~/.singularity` directory does not exist to creat it you can do: `singularity cache clean` * Pull `fastqc` ```singularity pull --name fastqc-0.11.9--0.sif https://depot.galaxyproject.org/singularity/fastqc:0.11.9--0``` * Config file: info about mount volume also findable there `/usr/local/etc/singularity/singularity.conf` * environment variable SINGULARITY_BIND or SINGULARITY_BINDPATH * From https://docs.sylabs.io/guides/3.0/user-guide/bind_paths_and_mounts.html Use this image for exercise 0-2 singularity pull hello-world.sif shub://vsoch/hello-world ### VI editor cheat Sheet > https://www.atmos.albany.edu/daes/atmclasses/atm350/vi_cheat_sheet.pdf ### Exercise 0 What do you notice about the ownership of files in a container started from the hello-world image? (e.g. take a look at the ownership of files in the root directory (/)) > #### Answer 0.1 : owned by root ``` Singularity> cd / Singularity> ls -lah total 45K drwxr-xr-x 1 training training 60 Oct 26 09:32 . drwxr-xr-x 1 training training 60 Oct 26 09:32 .. -rw-r--r-- 1 root root 0 Jul 19 2019 .coredone lrwxrwxrwx 1 root root 27 Jul 17 2019 .exec -> .singularity.d/actions/exec lrwxrwxrwx 1 root root 26 Jul 17 2019 .run -> .singularity.d/actions/run lrwxrwxrwx 1 root root 28 Jul 17 2019 .shell -> .singularity.d/actions/shell drwxr-xr-x 5 root root 127 Jul 19 2019 .singularity.d lrwxrwxrwx 1 root root 27 Jul 17 2019 .test -> .singularity.d/actions/test drwxr-xr-x 2 root root 1.7K May 15 2019 bin drwxr-xr-x 2 root root 3 Apr 10 2014 boot drwxr-xr-x 15 root root 4.1K Oct 26 07:02 dev lrwxrwxrwx 1 root root 36 Jul 17 2019 environment -> .singularity.d/env/90-environment.sh drwxr-xr-x 61 root root 2.1K Jul 19 2019 etc drwxr-xr-x 1 training training 60 Oct 26 09:32 home drwxr-xr-x 12 root root 241 May 15 2019 lib drwxr-xr-x 2 root root 43 May 15 2019 lib64 drwxr-xr-x 2 root root 3 May 15 2019 media drwxr-xr-x 2 root root 3 Apr 10 2014 mnt drwxr-xr-x 2 root root 3 May 15 2019 opt dr-xr-xr-x 226 root root 0 Oct 26 07:01 proc -rwxr-xr-x 1 root root 46 Jul 19 2019 rawr.sh drwx------ 2 root root 46 Jul 19 2019 root drwxr-xr-x 8 root root 141 Jul 19 2019 run drwxr-xr-x 2 root root 2.1K Jul 19 2019 sbin lrwxrwxrwx 1 root root 24 Jul 17 2019 singularity -> .singularity.d/runscript drwxr-xr-x 2 root root 3 May 15 2019 srv dr-xr-xr-x 12 root root 0 Oct 26 07:01 sys drwxrwxrwt 9 root root 40K Oct 26 09:33 tmp drwxr-xr-x 10 root root 150 May 15 2019 usr drwxr-xr-x 11 root root 160 May 15 2019 var ``` > #### Answer 0.2: ``` drwxr-xr-x 2 nobody nogroup 0 May 15 2019 bin drwxr-xr-x 2 nobody nogroup 0 Apr 11 2014 boot drwxr-xr-x 21 nobody nogroup 4080 Oct 26 09:25 dev lrwxrwxrwx 1 stefaan stefaan 36 Oct 26 11:39 environment -> .singularity.d/env/90-environment.sh drwxr-xr-x 61 nobody nogroup 0 Jul 19 2019 etc drwxr-xr-x 3 stefaan stefaan 60 Oct 26 11:39 home drwxr-xr-x 12 nobody nogroup 0 May 15 2019 lib drwxr-xr-x 2 nobody nogroup 0 May 15 2019 lib64 drwxr-xr-x 2 nobody nogroup 0 May 15 2019 media drwxr-xr-x 2 nobody nogroup 0 Apr 11 2014 mnt drwxr-xr-x 2 nobody nogroup 0 May 15 2019 opt dr-xr-xr-x 390 nobody nogroup 0 Oct 26 09:25 proc -rwxr-xr-x 1 nobody nogroup 46 Jul 19 2019 rawr.sh drwx------ 2 nobody nogroup 0 Jul 19 2019 root drwxr-xr-x 8 nobody nogroup 0 Jul 19 2019 run drwxr-xr-x 2 nobody nogroup 0 Jul 19 2019 sbin lrwxrwxrwx 1 stefaan stefaan 24 Oct 26 11:39 singularity -> .singularity.d/runscript drwxr-xr-x 2 nobody nogroup 0 May 15 2019 srv dr-xr-xr-x 13 nobody nogroup 0 Oct 26 11:28 sys drwxrwxrwt 20 nobody nogroup 420 Oct 26 11:36 tmp drwxr-xr-x 10 nobody nogroup 0 May 15 2019 usr drwxr-xr-x 11 nobody nogroup 0 May 15 2019 var ``` ### Exercise 1 In this container, try editing (for example using the editor vi which should be available in the container) the /rawr.sh file. What do you notice? If you’re not familiar with vi there are many quick reference pages online showing the main commands for using the editor, for example this one. > #### Answer: File was read-only; editing was not possible ### Exercise 2 In your home directory within the container shell, try and create a simple text file. Is it possible to do this? If so, why? If not, why not?! If you can successfully create a file, what happens to it when you exit the shell and the container shuts down? > #### Answer: you could create a file which is owned by training user. When you exit the singularity shell, it is still there. From host: ``` $ ls -lah ~/singularity total 313M drwxrwxr-x 2 training training 4.0K Oct 26 08:48 . drwxr-xr-x 17 training training 4.0K Oct 26 09:33 .. -rwxrwxr-x 1 training training 253M Oct 26 08:37 fastqc-0.11.9--0.sif -rw-rw-r-- 1 training training 0 Oct 26 08:48 file1.txt -rwxrwxr-x 1 training training 60M Oct 26 08:38 hello-world.sif ``` ### Exercise 3 Try to run the fastqc command by using the fastqc singularity image from the Galaxy depot on both example fastq files from day 1. > #### Answer (also works for HPC; small files): `singularity exec fastqc-0.11.9--0.sif fastqc ~/data/ecoli_*gz` > #### Daria: doesn't work `singularity exec -B ~/data:/scratch fastqc-0.11.9--0.sif ls ./scratch/` > #### Daria: works `singularity exec -B ~/data:/scratch fastqc-0.11.9--0.sif ls /scratch/` > #### Teresa: Answer does not work `singularity exec -B ~/datatest:/scratch sif-images/fastqc-0.11.9--0.sif fastqc /scratch/*.gz` glob pattern * is evaluated before running singularity, so scratch is not mounted yet > #### works: `singularity exec -B ~/datatest:/scratch sif-images/fastqc-0.11.9–0.sif bash -c "fastqc /scratch/*.gz` `singularity exec -B ~/datatest:/scratch sif-images/fastqc-0.11.9–0.sif fastqc /scratch/ecoli_1.gz` > #### Answer with -B: ``` $ singularity exec -B ~/containers/data/:/scratch ~/singularity/fastqc-0.11.9--0.sif fastqc /scratch/ecoli_1.fastq.gz WARNING: Skipping mount /usr/local/var/singularity/mnt/session/etc/resolv.conf [files]: /etc/resolv.conf doesn't exist in container perl: warning: Setting locale failed. perl: warning: Please check that your locale settings: LANGUAGE = (unset), LC_ALL = (unset), LANG = "C.UTF-8" are supported and installed on your system. perl: warning: Falling back to the standard locale ("C"). Started analysis of ecoli_1.fastq.gz Approx 5% complete for ecoli_1.fastq.gz Approx 10% complete for ecoli_1.fastq.gz Approx 15% complete for ecoli_1.fastq.gz Approx 20% complete for ecoli_1.fastq.gz Approx 25% complete for ecoli_1.fastq.gz Approx 30% complete for ecoli_1.fastq.gz Approx 35% complete for ecoli_1.fastq.gz Approx 40% complete for ecoli_1.fastq.gz Approx 45% complete for ecoli_1.fastq.gz Approx 50% complete for ecoli_1.fastq.gz Approx 55% complete for ecoli_1.fastq.gz Approx 60% complete for ecoli_1.fastq.gz Approx 65% complete for ecoli_1.fastq.gz Approx 70% complete for ecoli_1.fastq.gz Approx 75% complete for ecoli_1.fastq.gz Approx 80% complete for ecoli_1.fastq.gz Approx 85% complete for ecoli_1.fastq.gz Approx 90% complete for ecoli_1.fastq.gz Approx 95% complete for ecoli_1.fastq.gz Approx 100% complete for ecoli_1.fastq.gz Analysis complete for ecoli_1.fastq.gz ``` From host: ``` $ ls -lah ~/containers/data/ total 7.5M drwxrwxr-x 2 training training 4.0K Oct 26 10:03 . drwxrwxr-x 5 training training 4.0K Oct 24 13:50 .. -rw-r--r-- 1 training training 3.3M Jan 29 2021 ecoli_1.fastq.gz -rw-rw-r-- 1 training training 600K Oct 26 10:03 ecoli_1_fastqc.html -rw-rw-r-- 1 training training 344K Oct 26 10:03 ecoli_1_fastqc.zip -rw-r--r-- 1 training training 3.3M Jan 29 2021 ecoli_2.fastq.gz ``` #### Activity - To be pulled (pay attetion in the folder you will pull it) `singularity pull python-3.9.6.sif docker://python:3.9.6-slim-buster` #### Using it remotly singularity exec docker://ncbi/blast:2.10.1 blastp -version #### Sandbox ```singularity build --sandbox ./sandbox docker://ubuntu:18.04``` ```singularity exec ./sandbox bash -c 'apt-get update && apt-get install python'``` https://elearning.bits.vib.be/courses/introduction-to-docker/lessons/building-singularity-images/topic/using-singularity-recipes/ ![](https://i.imgur.com/bghPvZx.png) ![](https://i.imgur.com/K7Y86Xv.png) #### Syntax highlighting of Singularity definition in VScode https://github.com/onnovalkering/vscode-singularity example with debootstrap install it in the VM template sudo apt-get install debootstrap

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.