# Distributed Systems, Project 1

## Team Strong Dogs: Aidar Garikhanov and Insaf Safin

## Introduction
The task is to work with containerization and orchestration. Also we will work with container migrations and write web application to monitor weather in the cities all over the world. After developing the app we are going to deploy it using Docker Swarm.

## Understanding Docker-machine and Docker Swarm deployment

### 1. What is Docker-machine and what is it used for?
Docker-Machine is a tool that lets you install Docker Engine on Virtual Hosts. We can manipulate physical and virtual hosts using docker-machine commands: stop, check, restart hosts, transwer files to them.

### 2. What is Docker Swarm, what is it used for and why is it important in Containers Orchestration?
Docker swarm is a container orchestration tool, that makes a cluster from a group of phisical and virtual machines, that run docker.
It is used to deploy application services on multiple machines securely and efficienly. It is important in Containers Orchestration because it provides networking between hosts and scalability of services.

### 3. Install Docker-machine based on your virtualization platform (VirtualBox, Hyper-V, VMware), create a Machine (named Master), and collect some relevant information for you
As it can be seen from the screen, we use:
- Microsoft HyperV driver
- Boot2docker provisioner


### 4. Create two Workers as well. Later we will connect them into one swarm. Make a screenshot for docker-machine ls command. You should have 3 running machines.



## Container Docker Cluster farm deployment

### 5. Now that Docker Swarm is enabled, deploy a true container cluster farm across many Dockerized virtual machines. (One master and two workers). Verify the Docker Swarm status, identify the Master node(s), and how many workers active exist. Take as many screenshots as you need to explain the process.
- Connect to the master node via SSH
- Initialize a swarm
- Copy the command for workers
- Connect to worker 1 node via SSH
- Join the created swarm using the copied command
- Do the same steps as for worker 2
- Switch to master node
- Print the information about the swarm
- Print the information about the swarm using `docker info`
Status:
- Swarm is active
- 2 active workers exist

### 6. How can a Worker be promoted to Master and vice versa?
There are special commands:
- **docker node promote NODE**
- **docker node demote NODE**
These commands must be executed from manager nodes, promote makes manager from worker NODE, demote makes worker from manager NODE.
The requirements is that last manager NODE can not me demoted.

### 7. Deploy a simple Web page, e.g Nginx, showing the hostname of the host node it is running upon, and validate that its instances are spreading across the servers previously deployed on your farm.

We used simple python HTTP server for the base of our docker image. Here is is:

FROM python:3.8-alpine

EXPOSE 80

CMD cat /etc/hostname > index.html && python3 -m http.server 80

We build an image and push it to DockerHub.
On the master node we ran a new service with 3 replicas. To make the server accessible from the local host we opened port 80 as 80. To make the server display the hostname of the node instead of the container we binded a mount with a file containing hostname of the node (`/etc/hostname`). As a docker image we use the one we build.
Here is the result

### 8. How to scale instances in the Docker Swarm? Could it be done automatically?
We can scale instances using the following command:
`docker service scale <SERVICE-ID>=<NUMBER-OF-TASKS>`
There is no built-in autoscaling in Docker Swarm. However, we can use an external tools for load balancing.

## Docker Container migration - Application Distribution

### 9. Validate that when a node goes down a new instance is launched. Show how the redistribution of the instances can happen when the dead node comes back alive.

Drain worker1 and check how task are redestributed
The swarm manager maintains the desired state by ending the task on a node with Drain availability and creating a new task on a node with Active availability.
Let's make worker1 active again and check if distribution has been changed.
As we can see the destribution is the same (worker2 has 2 tasks).

### 10. Perform some update in your application, a minor change in your sample application for example. How to replicate the changes in the rest of the farm servers?

We can replicate the changes using `docker service update`
Let's change the image so the server display `Host: ` before the hostname.
Here is the result:

### 11. It is a good practice to monitor performance and logs on your servers farm. How can this be done with Docker Swarm? Could it be just CLI or maybe GUI?
It's always a good practice to monitor performance and logs.

We can check logs for Docker Swarm using `docker service logs`:
(it is outdated error message when we were trying to update services with a broken image)

There are a lot of GUI tools (mostly web apps) for Docker Swarm to monitor performance, for example:
- Prometheus - store and collect metrics
- Graphana - visualize metrics
- cAdvisor - monitor resouce usage and performace characteristics
- NodeExporter - collector for node/hosts metrics
- Alertmanager - send message to communication channels

## Playing with Memory

### 12. Please explain what is “Out Of Memory Exception (OOME)”, how it could affect Docker services, and which configuration can be set to avoid this issue?
This exception is raised when there is not enough memory to execute all processes. In such case some of the processes is killed.
Such behaviour can shut down all the docker services if Docker daemon is killed.
To avoid that one can assign docker daemon higher proirity that containers. So some container will be killed first. Docker Daemon can restart those containers when memory is released.
Another option is to limit memory given to each container.

### 13. Deploy a docker container with at least 15% of CPU every second for memory efficiency.

We can limit the CPU usage of a container using flags:
- `--cpus` in we deploy a container using `docker-engine` (just `docker run`)
- `--limit-cpu` if we deploy a container in `docker swarm` (`docker service start`)
Let's see how to limit CPU usage on Docker Swarm. We run a docker image with a stress tool on 3 replicas.
Here is the CPU load:
Now we limit CPU usage (50%):
and with 15% CPU bound the result is following:
As we can see, Windows task manager display the actual CPU load of virtual machines, but it is not the same as CPU load of the docker container. As we decreased the CPU limit, CPU usage of virtual machine decreased proportionally as expected.

## Compression

### 14. Verify the size of the Docker images that you're working with. Can this size be reduced and how can we achieve this?

The first and most significant choise is using light base image. We use *python:3.8-alpine*. It is recommended to use it because its weight is much less than ubuntu or common python images.

We minimised the number of layers in the docker image. For that we combined all similar commands to one layer. For example combined several RUN commands in Dockerfile to one RUN command.

Also we could delete all temporarily needed programs in the same layer we install them, so they would not be added to the image at all! But we dont have such cases.

Moreover, we could use  no-install-recommends on apt-get install. So no redundant programs would be installed. But during istallation we use only pip, so this is not the case. After all optimisations we have image of this size.


## Web application

### 15.Instead of a simple web page in step 7 elaborate your own web application. Be creative. Again, interaction with a user is required.

#### We implemented an interactive web application for monitoring weather all over the world.
Our app is based on Django web framework.

#### One can add new city to the list or delete it.

#### After adding Sydney and deleting London and Oslo

#### Overal view and pop-down menu

#### Deployment
We pushed the docker image to Docker Hub and cloned the git repository on the master node and deployed the stack
The app is accessible via master IP address:


Why our app is scalable?
First, we use external OpenWeather API. It is free and has the limit on the number of calls to it. We cache the results we get from a web API for 5 minutes and store them in sqlite database. In production we can use redis as a separate service in Docker Stack instead of sqlite database.
Second, we also use sqlite as a database to store cities. If we want to expand the app, we can run PosgreSQL database as a separate service.

#### Code can be found at https://github.com/a1d4r/weather_app

## Conclusion
During this project we learned how to work with docker-machine, docker swarm, hyperV and docker stack. We had great experience of implementing web application on django. We experienced problems connected with Windows OS, but solved them all. Overall, we learned a lot from this project and think that it gave us worthy knowledge.