:::success
# LS Lab 2 - Container Orchestration & Load-balancing
**Authors: Alexandr Ismatov, Fige Polina**
:::

## Task 1 - Choose Container Engine & Orchestration

1. Choose a container engine. Some suggestions: **Docker**

:::info
Deploy a true container cluster farm, across several team machines. It is however recommended to proceed in a virtual machine environments so the worker nodes can have the exact same system patch level (which makes it easier).
*Bonus: if you choose Docker, play with alternate storage drivers e.g. BTRFS or ZFS instead of OverlayFS*
:::

We decided to use Docker as a container technology. First of all, we installed the required docker packages on one machine with the following commands:

# sudo apt install apt-transport-https ca-certificates curl software-properties-common
# curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
# sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable"
# sudo apt install docker-ce

Then we add our current user to the **`docker group`** to make able run the **`docker`** command without root privileges:

# sudo usermod -aG docker ${USER}
# su - ${USER}

After getting the Docker installed on one VM, we cloned the VM and changed the hostnames. The main issue here is the IP addresses. Somehow all cloned machines were getting the same IP address even if their MAC addresses were different. After some investigations, we found that the problem was in **`/etc/machine-id`** file. For all cloned VM's, it was the same, which is why the same IP was provided. After changing the machine-id in each VM, they started to get unique IP addresses.

:::info
***Bonus:** if you choose Docker, play with alternate storage drivers e.g. BTRFS or ZFS instead of OverlayFS.*
:::

For example, take ZFS, a file system that creates pools for storing information. To do this, we will create another **"Ubuntuzfs"** virtual machine, install docker on it and temporarily stop it while we configure zfs.

#stop docker service
sudo systemctl stop docker

#install zfs-package
sudo apt install zfsutils-linux

#create a backup and clean the docker folder (also check it manualy)
sudo cp -au /var/lib/docker /var/lib/docker.bk
sudo rm -rf /var/lib/docker/*

#creating a pool on the required device
sudo zpool create -f zpool-docker -m /var/lib/docker /dev/vdb 

<center>

Figure 1 - Disks output
</center>

In our case, there is only one additional storage, we added it via Virt-manager in the settings for adding a hard drives. We create a pool on the required device, in this case, the [Docker's guide](https://docs.docker.com/storage/storagedriver/zfs-driver/) to using this file system and the [Oracle manual](https://docs.oracle.com/cd/E19253-01/819-5461/gaynr/index.html) itself provides a choice on how many devices to create a pool. There are mirror versions, there are striping, hybrid ones, the only difference is in the final access to the recorded data.

<center>

Figure 2 - Zpool
</center>

Now you need to edit the `/etc/docker/daemon` and install `zfs` as the storage driver.

<center>

Figure 3 - Daemon.json
</center>

#launch the docker
sudo systemctl start docker

#and look at the information
docker info

<center>

Figure 4 - ZFS Docker storage
</center>

:::info
2. Сhoose an orchestration engine then: **Docker Swarm**
:::

Docker Swarm doesn't require any additional setup, it comes automatically with **`docker-ce`** package. To start it we need to run a **`docker swarm init`** command:

<center>


Figure 5 - Swarm init
</center>

As we can see from the screen, the VM was assigned in a Manager role, and it gives the command with the token that we can use in Worker's VM to add them in a group:

<center>



Figures 6,7 - Joining workers
</center>

Similarly, using a token, we attach virtual machines to another workstation (**st7**). We configured the Docker for them following the commands described above. And functioning in a shared subnet allowed you to connect to them without problems. Now our farm contains 1 Manager and 4 Workers.

<center>

Figure 8 - Nodes status
</center>

:::info
3. Analyze and describe the best practices of usage the chosen container engine
:::

Perhaps the best practice in using Docker is to create containers from scratch, without a distribution kit. The fact is that most ready-made Linux images include a large number of pre-installed packages, some of them may be vulnerable, and most of them are simply useless, plus, images without a distribution kit are much easier. An example would be [Distroless](https://github.com/GoogleContainerTools/distroless) images, they contain only a minimal set of libraries (glibc , libssl и openssl) needed to run Python or other frameworks.

Another best practice is to use the container using user rights, not root (UID 0). Although this practice is not observed in [58% of cases](https://sysdig.com/blog/sysdig-2021-container-security-usage-report/) (and we are among them). It is good because the service or application will have access only to the information that is necessary for their work, nothing more. However, all executable files and the ability to modify them should be given to root, even if it is executed by a non-root user and should not be world-writable.The application user only needs the rights to execute the file, not the ownership. Thus, it can protect applications from attacks related to access abuse and overwriting.

Also, if we talk about security, it is better not to leave open port 22 for SSH. Yes, we use it now in the lab because it is convenient, but from a security point of view, we need to use only those ports that are necessary for the application to work.

And another good practice is to use tools to detect bad practices. For example, [Haskell Dockerfile Linter (hadolint)](https://github.com/hadolint/hadolint) can detect bad practices in Dockerfile and identify problems inside shell commands executed by the RUN instruction.

## Task 2 - Distributing an Application/Microservices

:::info
Base level (it means that this task will be evaluated very meticulously): deploy at least a simple application (e.g. a simple web page showing the hostname of the host node it is running upon) and validate that its instances are spreading across the farm.
:::

In the beginning nothing was running in our containers:

<center>

Figure 9 - Inside empty services
</center>

At a base level we installed NGINX on one machine: **`docker service create --name nginx-web --publish 8080:80 nginx`**

<center>

Figure 10 - New service
</center>

and than scaled it to all three VM: **`docker service scale nginx-web=3`**

<center>

Figure 11 - Scale service
</center>

#### Shared Folder

To make Docker Swarm able to share folders we used GlusterFS. The first step was to edit **`/etc/hosts`** file in all VM connected to the Swarm

<center>

Figure 12 - /etc/hosts
</center>

The next step was to install the GlusterFS on all stations, run it and make a working directory for it:

# sudo apt install -y glusterfs-server
# sudo systemctl start glusterfs-server
# sudo mkdir -p /gluster/shared_volume

Additionally it requires a ssh-keys, so we generate a pair for each station with: **`ssh-keygen -t rsa`** command.

After, on the Manager station we add all worker nodes into a pool with:

# sudo gluster peer probe docker-worker1 
# sudo gluster peer probe docker-worker2 
# sudo gluster peer probe docker-worker-polina 
# sudo gluster peer probe docker-worker-zfs-polina

and verified the result with: **`sudo gluster pool list`** command:

<center>

Figure 13 - full pool list
</center>

Time to connect them all with:

# sudo gluster volume create staging-gfs replica 3 \
docker-manager-alex:/gluster/shared_volume \
docker-worker1:/gluster/shared_volume \
docker-worker2:/gluster/shared_volume force

After all configuration has been finishe, we started the share: **`sudo gluster volume start staging-gfs`**.

The last step was to mount the shared folder and make it permanent for next logins:

# sudo echo 'localhost:/staging-gfs /mnt glusterfs defaults,_netdev,backupvolfile-server=localhost 0 0' >> /etc/fstab
# sudo mount.glusterfs localhost:/staging-gfs /mnt
# sudo chown -R root:docker /mnt

To check that we did everything right: **`df -h`** command:

<center>




Figures 14, 15, 16 - Checkup
</center>

Now, we are ready to start a container that will take place in a shared folder with: **`docker service create --name nginx --mount type=bind,source=/mnt/nginx,target=/usr/share/nginx/html --publish 8080:80 nginx`** command.

<center>


Figures 17, 18 - Create a container
</center>

And to be sure that all stations have the same folder and files, we run a simple **`ls`** command:

<center>



Figures 19, 20, 21 - Checkup
</center>

## Task 3 - Validation
:::info
Validate that when a node goes down a new instance is launched. Show how the redistribution of the instances can happen when the broken node comes back alive. Describe all steps of tests in your report in details.
:::

In case of failure of a node whose containers were involved, swarm will detect that the desired state does not match the actual one and will automatically correct the situation by redistributing containers to other available nodes.

Initially, we have already tested the scalability of services on all 5 hosts, but now, to demonstrate fault tolerance, we will reduce the number of replicas of services to 3.

<center>

Figure 22 - Preparing services
</center>

Now we have one container on each node, now we will disable, for example, the "ubuntuzfs" node.

#run on ubuntuzfs
docker swarm leave  

To be honest, we didn't even have enough time to notice the change in the number of nodes on which the application is running. However, the final table shows that as soon as the Ubunuzfs node failed, Swarm raised the service on a new free Docker-w2 node:

<center>

Figure 23 - New instance is launched
</center>

Simply leaving the node from the cluster does not change anything, it should have been deleted beforehand, because in this case the Svarm is waiting for it to return, but after restoring the node's operability, the application has to be restarted. Then you can see in the output that the process on the backup node went into the archive and ended, but it rose again on the restored ubuntuzfs node.

#for node
docker node rm <node>

#the dispatcher stops the tasks running on the node
docker node update --availability=drain <node>

<center>

Figure 24 - Result
</center>

But there is another option, if a node suddenly leaves the swarm, another node will take its place. Then, if the first node is restored, you can suspend the distribution of tasks for the replicant node, in which case the lost node will return to the task again. If you then return the activity to the additional node again, then it will no longer be included in the task.

<center>

Figure 25 - Other test (including processes for nodes removed by force)
</center>

> If the last node of the manager leaves the swarm, the swarm becomes unavailable, which requires disaster recovery measures

In this case, we will appoint another node as a manager in order to create a quorum and see what happens when the manager leaves the cluster.

<center>

Figure 26 - New manager
</center>

> The reason why Docker swarm mode is using a consensus algorithm is to make sure that all the manager nodes that are in charge of managing and scheduling tasks in the cluster, are storing the same consistent state.

> Having the same consistent state across the cluster means that in case of a failure, any Manager node can pick up the tasks and restore the services to a stable state. For example, if the Leader Manager which is responsible for scheduling tasks in the cluster dies unexpectedly, any other Manager can pick up the task of scheduling and re-balance tasks to match the desired state.

<center>

Figure 27 - An example of how not to do
</center>

But if this has happened, then there is an opportunity to "reach quorum" with the help of a command running on the manager, which appoints him as the only manager in the cluster:

docker swarm init --force-new-cluster --advertise-addr enp1s0:2377

I will try this a little later, but for now I will show how the manager is reassigned after the current leader leaves the network. As you can see, cluster management has received the next available node with the manager role.

<center>

Figure 28 - New leader
</center>

And now, if you go back to the command `--force-new-cluster`, according to the documentation, it allows you to return the manager to the swarm without losing working services, networks, etc. However, despite multiple re-creation of the swarm, attempts to delete the manager and restore it, I never managed to see this command in operation. Each time it is a completely new swarm that does not store any data about past services. Can you explain in the comments to the report why it is not possible to keep the swarm working when using this command?

## Task 4 - Load-balancing

By default Docker Swarm has a Load Balancing feature. It starts from the deployment point when it automatically distributes the replicas over all nodes. In addition, a Layer 7 Load Balancer such as NGINX can be implemented. It has two modes, a free Basic one and paid Plus.

We are going to install and use the Basic one in our experiments.

In the beginning, we need to prepare a network that we are going to use with: **`docker network create -d overlay loadbalancing`** command

<center>

Figure 29 - Prepare network
</center>

Then we add a new container named hello over all our nodes: **docker service create --name lb --network loadbalancing --publish 8080:80 --replicas 5 nginxdemos/hello**

To make a proper Load Balancing with NGINX, we modified a few files on the standard repository.

The first one is the default.conf file:
<center>

Figure 30 - default.conf
</center>

The second one is the index.html file:

<center>

Figure 31 - index.html
</center>

And finally, the Dockerfile itself, where we deleted original files and replaced them with modified ones.

<center>

Figure 32 - Dockerfile
</center>

Now, we are ready to build a new image with **`docker build -t nginx_lb .`** command.

<center>

Figure 33 - Build a new image
</center>

Our local image is ready, now to make it available for Docker Swarm distribution, we upload it into our own DockerHub repository

<center>

Figure 34 - DockerHub repository
</center>

The last step is to create a service on one workstation and test how the results: **`docker service create --name lb --network loadbalancing --publish 8090:80 aismatov/inno`**

<center>

Figure 35 - Create a service and test
</center>

Working Demo is uploaded as a video file.

## References:
1. [ZFC for Docker](https://docs.docker.com/storage/storagedriver/zfs-driver/)
2. [Oracle ZFS](https://docs.oracle.com/cd/E19253-01/819-5461/index.html)
3. [Dockerfile: Best practices](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/)
4. [--force-new-cluster](https://github.com/docker/for-linux/issues/523)
5.