# Tutorial of Singularity

**This document is produced by Yi-Ping Huang, who is a newbie in the container busniess. If you found mistakes or have suggestions, please send me an email. I will appreciate your help to make this document better:)**

## What is the concept of container in plain language?

We as developers will like to write a program. The procedure to write a program is similar to bake a cake.

To bake a cake, we need to be in a suitable environment, a kitchen. Usually, we might rent an apartment where we don't have full permission to remodel our kitchen. But that's fine, we can still arrange our tools and ingredients as we want. We might put some sugar on the table, an egg scrambler in the cabin, and so on. After years of development, we finally find the magic recipe such that we can bake a perfect cake.

Now we want to tell people how to do that and reproduce our perfect cake. Imagine what we need to do. We might need to call our friends and teach our friends the magic recipe. The first difficulty we will encounter is: we don't know where are the sugar and milk. Our friend might also forget where the milk is. If our friend does have sugar and milk, it might not be the kind of sugar or milk that we need to create our perfect cake. This is the issue of dependency. In addition, we might need to face another problem that the oven is not the one that we are familiar with. However, the landlord won't give our friend permission to remodel the kitchen and install the correct oven for our perfect cake. So our friend might need to go through a ridiculous process to use extension cords to install the correct oven in the bedroom of our friend. With a bag of sugar that we don't know how sweet it is, a box of milk that we don't know whether it is already expired, and an oven with an unstable power supply sitting in the wrong environment. It seems very unlikely that we can reproduce the perfect cake under such a situation. We might eventually get something that looks like a perfect cake but with awful taste.

To solve that problem, one way is to rebuild our friend's house and reproduce all the environment including the living room and the kitchen. Putting exactly the same kind of sugar and milk at exactly the same spot of our kitchen inside your friend's house. This is the idea of the virtual machine.

The idea of the container is to create a magic suitcase(Like Moody's trunk in Harry Potter) that you can make an exact copy of your kitchen and bring to your friend's house. All you need to do is to make sure that you can teach your friend how to use this magic suitcase. Since the magic suitcase contains sugar, milk even the correct oven, it is guaranteed to reproduce the perfect cake.

## Why singularity?

* **A Program could have dependency issue**: We usually develop programs on a machine that we are familiar with. So we could install a library in a directory that is different from the default case. We might use some files that are only contained in an old version of certain Linux distribution. When we move our program and try to run it on a different computer, it might cause big trouble. For example, one needs to find a suitable path to link your program to the library. The different versions of the libraries could have subtle settings at the linking stage. The different machines could also mean a different OS, some files might be missing in the newer version of the distribution. That makes it challenging to have a reproducible result at different space-time points.
* **We need a portable snapshot of the developing environment!** This can be achieved by the virtual machine. But a more advanced technique is the concept of containers! There are several different realizations for such a concept. **Docker** and **singularity**. The advantage of using containers is that it is much efficient compared with the virtual machine. To start, one can learn how to use **Docker**. However, to work on the HPC, we prefer to learn **singularity** which has a more compatible security model comparing with the HPC architecture.
* **We can pack anything we want into the container locally and take that with our source code to anywhere with singularity installed.**
* Using singularity, we will have reproducible results that help us finding possible mistakes/bugs. We learn from our past to achieve bigger things in our future.
* **The workflow to run a program on the HPC**:
1. We develop our program locally and solve the dependency issue with **singularity**.
2. Use **singularity** to create an image that can run the program on the HPC.
3. Send the image file to our HPC and run it!

### Why do we need singularity?

**To have reproducible results! Even after our cluster is destroyed physically or virtually! Nothing can stop our research project or graduation plan. Nothing!**

## Install Singularity locally

On our own machine, we have the highest permission. We can follow the instruction to install singularity locally.

*update*: The new version of singularity is called *apptainer* one can install that easily on rockylinux.

## Simplest working example: Diagonalizing a matrix using armadillo linking with MKL library

We will write a simple c++ program that diagonalizes a matrix and outputs the corresponding eigenvalues and eigenvectors using the armadillo and MKL library.

### Construct the singularity container with the suitable developing environment

#### The base environment using the definition file

The definition file is the file that defines our container.

For example, suppose we want to start with a clean ubuntu20.04 system, we can configure that in our ```Trunk_of_Moody.def``` definition file as

Bootstrap: docker
From: ubuntu:20.04

This means we want to pull it from the docker image library and the container is a clean ubuntu20.04 system.

> 1. The source of the image file could come from the docker or singularity library. It seems I will hit some problem if I pull it from the singularity library. However, it seems pulling from docker can make things work. So let's just use the working case for now.
> 2. One can also try different OS, for example, centos:8, alpine:3, *e.t.c.*)

The setting of the timezone of the container also needs some tweaks. In the definition file, we put

%post
    export DEBIAN_FRONTEND=noninteractive
    apt-get -y update
    apt-get install -y --no-install-recommends tzdata
    ln -sf /usr/share/zoneinfo/Asia/Taipei /etc/localtime

> 1. Commands in the ''%post'' section are executed within the container after the base OS has been installed at build time.
> 2. This part of the code sets the timezone to be in Taipei.

Suppose we need the following package to be installed from the package manager.
* vim
* build-essetial
* libarmadillo9
* libarmadillo-dev
* wget
We can use ```apt-get -y install``` to install those packages. So we keep putting them in the ```%post``` section to have

%post
    export DEBIAN_FRONTEND=noninteractive
    apt-get -y update
    apt-get install -y --no-install-recommends tzdata
    ln -sf /usr/share/zoneinfo/Asia/Taipei /etc/localtime
    apt-get -y install vim
    apt-get -y install build-essential
    apt-get -y install libarmadillo9
    apt-get -y install libarmadillo-dev
    apt-get -y install wget
    unset DEBIAN_FRONTEND

Those are packages that one can get directly from the package managers. So we can simply include them in the .def file. For ```rpm``` system we can use a similar definition file as

Bootstrap: docker
From: centos:8

%post
	dnf install -y tzdata
	ln -sf /usr/share/zoneinfo/Asia/Taipei /etc/localtime
    dnf install -y vim
	yum install -y epel-release
	dnf install -y 'dnf-command(config-manager)'
	yum config-manager --set-enabled powertools 
	dnf install -y gcc-c++
	dnf install -y armadillo
	dnf install -y armadillo-devel

The little difference is the epel-release repo needs to be installed and powertools needs to be enabled if we want to use yum or dnf to install the tools we need.

*update*: Since CentOS is at the EOL. So we replace the def file by image of the *RockyLinux*. Also, powertools are replaced by crb.

Bootstrap: docker
From: rockylinux:9

%post
	dnf install -y tzdata
	ln -sf /usr/share/zoneinfo/Asia/Taipei /etc/localtime
    # The epel-release contains lots of packages which might be useful.
	yum install -y epel-release
    # The following lines are used for enable the "Code Ready Builder". It contains the required packages for armadillo, such as hdf5, lapack, etc.
	yum install -y dnf-plugins-core
    yum config-manager --set-enabled crb 
	# The following lines install the compiler and the armadillo libraries
    dnf install -y gcc-c++
	dnf install -y armadillo
	dnf install -y armadillo-devel

> Sometimes we need to install blas-dev or lapack-dev to link to the armadillo libraries correctly. Depends on the distribution of Linux.

Now we are ready to build our system. There are two ways to build the container, the sandbox, and the image.
1. Sandbox: we can build a sandbox where all the directories of the system are faithfully constructed, so we can modify our container as we want. (We need to set the appropriate permission level to do that.)

sudo singularity build Trunk_of_Moody.sif Trunk_of_Moody.def

2. Image: we can build an image where the container encapsulates everything that cannot be modified, so our developing environment becomes eternal.

sudo singularity build  --sandbox Trunk_of_Moody Trunk_of_Moody.def

In the sandbox form, since we have the full directories reproduced, we can actually install third-party packages as usual. Once we configure all the packages we need, we create the ```.sif``` image file to keep our environment. In that way, we can guarantee our program will give us reproducible results. We can also upload the image file to the cluster so we don't need to ask the system manager to install packages for us, our program can run in a self-contained way with the container.

Before we create the ```.sif``` image, let's try to install a third-party package. One of the most important ones is the intel oneAPI which contains the intel compiler and the optimized math kernel library (MKL).

#### Install MKL

Once we have our container in the sandbox form, we can enter our container as root and install the required packages.

sudo singularity shell --writable Trunk_of_Moody

To install intel-oneapi, please follow the [instruction](https://software.intel.com/content/www/us/en/develop/documentation/installation-guide-for-intel-oneapi-toolkits-linux/top/installation/install-using-package-managers/apt.html).

Now we are ready to check whether we have everything settled.

We can check who we are

In the default configuration, the directories ```$HOME```, ```/tmp```, ```/proc``` ,```/sys``` , ```/dev```, and ```$PWD``` are among the system-defined bind paths.

The default path to install intel-related package is ```/opt``` which is not mounted. So it will not conflict with the host intel packages. Once we install the intel packages successfully, we can find the setvars.sh file in ```. /opt/intel/oneapi/setvars.sh``` and set the variables accordingly.

We can also check whether the required software function properly.

#### Configuration of armadillo
To link to MKL properly, we need to edit the configure.hpp file of armadillo

Uncomment ```#define ARMA_BLAS_LONG_LONG```
Uncomment ```#define ARMA_USE_CXX11```
Comment out ```#define ARMA_USE_WRAPPER```
Comment out ```#define ARMA_USE_FORTRAN_HIDDEN_ARGS```

Find the lines that are required to be commented out.

### A simple program that diagonalize a matrix
We define a matrix $\boldsymbol{A}$ as
$$ \boldsymbol{A}=\left(\begin{array}{cccc} 0 &1 &2 &3\\ 1 &1 &2 &0\\ 2 &2 &4 &7\\ 3&0&7&-1\end{array}\right)\text{.}$$
And diagonalize the matrix using armadillo by linking it with **lapack** and **MKL**.

The eigenvalues of this matrix is $\{10.4303,-6.46169,0.868493,-0.837117\}$.

The coresponding eigenvectors are
$$\left(
\begin{array}{cccc}
-0.319114 & -0.193235 & -0.751575 & -0.544024 \\
\hline
-0.236278 & 0.17193 & -0.523305 & 0.800478 \\
\hline
-0.0466949 & -0.963282 & 0.0866866 & 0.249785 \\
\hline
0.916602 & -0.072028 & -0.392139 & 0.0296678 \\
\end{array}
\right)$$
We should see this result once our simple program runs correctly.