# Report. Configuration of CI/CD with Github Actions and AWS
**Authors:** Lada Morozova, Denis Schegletov, Danis Alukaev, Maxim Pryanikov.
**Group:** B19-DS-01, Innopolis University.

## Prerequisites
Doctorinna is an open-source application for determining the user's risk group for the widespread diseases by medical parameters. This application will allow people in the form of a survey without undergoing a medical examination to understand whether it is necessary to be examined by a doctor and change their lifestyle.

We are ambitious and intend to develop our [open-source API](https://github.com/Doctorinna/backend/tree/master) in order to analyze a variety of biomedical information completely free of charge. There is already provided a microservice for the analysis of biomedical information, but in the near future we will start developing tools for image processing!

## 1. Objectives
The main goal was to set up the continuous integration and deployment (CI/CD pipeline) for the [Doctorinna](https://github.com/Doctorinna) API in AWS infrastructure. In order to achieve it, we identified 4 main tasks to be performed:

1. Setting up the AWS infrastructure;
2. Configuring secure connection and load-balancing in Nginx;
3. Dockerizing the services inside our project: application, Postgres database, RabbitMQ message broker, Nginx web server;
4. Creating Github actions for CI/CD.

## 2. Execution plan
To complete the tasks and achieve our goal, our team followed the following steps.

1. Set up the Amazon infrastructure:
1. Amazon Virtual Private Cloud
- Create Public, Private, Database subnets
- Configure Internet Gateway
- Configure NAT Gateway
2. Set up EC2 instances
- Create 4 instances
- Configure static IP adresses
- Install Docker and Docker-compose
- Create firewall
- Improve securite of instance with public IP address
- Configure inbound rules
2. Configure Nginx server
- Generate certificate for HTTPS
- Configure Nginx to use SSL
- Configure load-balancing
5. Automate configuration
- Create a script to pull the latest Docker image in application and database instances
6. Set up Github Action
- Invoke script for automation of configuration on push
7. Deploy frontend
8. Load testing
- Test application with use of Yandex Tank
10. Delivery
- Create demo of application's work
## 3. Guidelines
This section consists of the exhaustive description of methodology we used and report about the performed over the resulting system tests. The methodology is compiled in the form of a plan that might be useful not only for system administrator of Doctorinna infrastructure, but also person aiming to reproduce the obtained results.

### 3.1. Build AWS Infrastructure
Amazon Web Services (AWS) was chosen as the cloud computing platform for our project. The privacy of our users is very important to us, so the choice of AWS complying FIPS 140-2 Level 2 certification is justified. In addition, the platform offers speed and flexibility for applications and in this section we will show how the infrastructure for our project has been configured.

The following figure shows the design of our infrastructure. All the computer resources are located inside the Amazon Virtual Private Cloud (VPC) with range of IP addresses 10.0.0.0/16.


To simplify the overall design there was used only one availability zone. Three subnets public, private, and database are entirely reside inside this availabity zone. Consider each subnet independently:
1. **Public Subnet (10.0.11.0/24)**
This subnet has an access to the internet through the Internet Gateway. It also keeps the NAT Gateway that we will discuss in next point. The subnet contains one publicly available Amazon Elastic Compute Cloud (EC2) server carrying the load balancing functions.

2. **Private Subnet (10.0.12.0/24)**
This subnet has an access to the internet through the NAT Gateway that we discussed recently. It contains two EC2 instance serving the application. The load balancer will distribute the requests among them.

3. **Database Subnet (10.0.13.0/24)**
This subnet does not has an access to the internet. Indeed the usage of completely isolated database subnet is a common practice in the industry. It contains one EC2 instance keeping the database management system (DBMS), and AMQP message broker.

### 3.1.1. Set up Amazon VPC
Amazon Virtual Private Cloud (VPC) is a cloud service that allows to design logically isolated virtual network. According to the design, the VPC has Classless Inter-Domain Routing (CIDR) block of 10.0.0.0/16.

Navigate to VPC service inside the AWS. Choose option "Create VPC", and set up the name tag and CIDR block as it is shown below.


The VPC should appear in the list of your VPCs. Make sure that all configurations coorrespong to one you specified.


### 3.1.2. Creating Internet Gateway
The Internet Gateway allows publicly available load-balancing server to have an Internet access. In order to attach it to the VPC navigate to "Internet Gateways" and click on "Create internet gateway". Set up the name tag.


Select recently created Internet Gateway, go to "Actions" and choose "Attach to VPC". Choose the VPC and save the configuration. The result should be as follows.


### 3.1.3. Creating Subnets
There should be created 3 subnets: public, private, and database. In order to create new subnets, navigate to "Subnets", and choose "Create subnet". Select VPC, and set up subnet name, CIDR block as follows.


Repeat the same procedure with private and database subnets. Obviously, the difference will be in the name, and also in the CIDR block: for public subnet it is 10.0.11.0/24, for private - 10.0.12.0/24, and for database - 10.0.13.0/24. As a result, our list should look similar to the following.


In public subnet the IP address should be auto-assigned. This configuration can be done in "Actions/Edit Subnet settings".


### 3.1.4. Connect Internet Gateway to public subnet
This step can be accomplished by configuration of Route table for the public subnet. Navigate to "Route tables", and choose "Create route table". Set up the name and select VPC. Add new entry 0.0.0.0/0 to the list of routes using "Edit routes". As a target select Internet Gateway.


### 3.1.5. Creating NAT Gateway
NAT Gateway allows instances from private subnet to access the internet. In order to configure it navigate to "NAT Gateways", and choose "Create NAT gateway". Set up the name, choose public subnet, and allocate elastic IP.


Similarly to 3.1.4 connect NAT Gateway to private subnet. The result should be as follows.


### 3.1.6. Adding the computer resources
As computational resources there are used EC2 instances. Navigate to "EC2", and choose "Launch instance". The procedure for all four instances is quite similar.

As an Amazon Machine Image select Ubuntu 20.04 LTS. As an instance type select t2.micro. On the third step make sure to choose necessary VPC and subnet.


Another sensitive moment is the selection of static IP. It can be done on the same (third) step in the "Network interfaces" section.


On the step 6, configure security group according to this table:
|Instance|Private IP|Used for|Port|
|---|---|---|---|
|Load balancer|10.0.11.10|HTTPS<br>HTTP<br>SSH |80<br>443<br>4596|
|Application 1 <br> Application 2|10.0.12.10 <br> 10.0.12.11|SSH<br>Django |22<br>8000|
|Miscellaneous|10.0.13.10|SSH<br>Postgre SQL<br>Rabbit MQ|22<br>5432<br>5672|

### 3.2. Configure EC2 instances
Once the infrastructure is ready, we need to configure the instances.

### 3.2.1. Install Docker and Docker-compose
Whole CI/CD pipeline is based on the usage of Docker containers. Therefore, firstly install Docker and Docker-compose. To do so, we write a 2 installation scripts:

Installation script for the Docker:

sudo apt-get update
sudo apt-get install \
    ca-certificates \
    curl \
    gnupg \
    lsb-release

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io
echo "$(sudo docker version)"

Installation script for the docker-compose:

sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
echo "$(docker-compose --version)"

Then we move the scripts from local machine to servers:

$ scp -i doctorinna-cluster.pem docker.sh ubuntu@18.216.251.167:~/
$ scp -i doctorinna-cluster.pem docker-compose.sh ubuntu@18.216.251.167:~/

After it we execute the shell scripts on server to install Docker and docker-compose:

#connect to instance
$ ssh -i doctorinna-cluster.pem ubuntu@18.216.251.167

#installing docker and docker-compose
$ sh docker.sh
$ sh docker-compose.sh

In similar way we transfered the scripts from server in public subnet to servers in private subnets and execute the scripts for all servers:






### 3.2.2. Making the connection secure

The objective of this section is the publicly available server 10.0.11.10. The goal is to secure the connection in such a way that it will not be a threat for Bruteforce attacks. Let's start with assigning names for each IP address in`/etc/hosts`:

# /etc/hosts
10.0.12.10 app1
10.0.12.11 app2
10.0.13.10 db

For ssh key-pair we have used algorithm ed25519. The rsa algorithm currently widely used is considered to be slower and less safe in comparison with ed25519. Moreover, the ed25519 public key only contains 68 characters (rsa 3072 has 544). The following command generates a key inno via ed25519 algorithm. The option -C is not required, but usually it is set to be my email:

$ ssh-keygen -t ed25519 -C "[e-mail]" -f inno

Then we login to the server from public subnet `ssh -i doctorinna-cluster.pem ubuntu@18.216.251.167` and add new user:

$ sudo adduser inno
$ usermod -a -G sudo inno
$ su - inno
$ sudo mkdir ~/.ssh
$ sudo chmod 0700 ~/.ssh
$ touch /home/inno/.ssh/authorized_keys
$ chmod 600 /home/inno/.ssh/authorized_keys
$ sudo -- sh -c "echo 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFrUxQdeRN1Vf/HvnaMiK40H5qdkWnjYe9r7Q1yIwHdH [e-mail]' > ~/.ssh/authorized_keys"
$ sudo chown -R inno:inno ~/.ssh

Next we made the connection more secure aiming to prevent the bruteforce attacks. For that purpose we change the ssh configuration file`sudo nano /etc/ssh/sshd_config`.

1. Add the line `Port 4596` to listen only to port 4596. However, it could be set to any number that is not allocated for different services and under 65535. In practice it makes sense to use port numbers greater than 1024. Otherwise, there is a higher chance to find it using the brute-force scanner. From this point on, optinon`-p` will be needed to connect to the VM.

2. Changed line `PasswordAuthentication yes` to `PasswordAuthentication no`. This change will prohibit authentication via password that is not secure and prone to brute-forcing. It is also recommended to add line `PermitEmptyPasswords no`.

3. Changed line `ChallengeResponseAuthentication yes` to `ChallengeResponseAuthentication no`. Some people find this option more secure and suggest leaving it on and `PasswordAuthentication off`. Their believe that attacking becomes more complicated to automate because the system can ask a set of several questions. Still, it generally asks only for password, so we would recommend turning it off.

4. Change line `PermitRootLogin yes` to `PermitRootLogin no`. Since the root user is capable of doing pretty much everything over the system, the attackers frequently try to authenticate under the root. Thus, it is better to disable such possibilities.

5. Add line `PubkeyAuthentication yes`. Following option allow the authentication using SSH keys. We would recommend using the ed25519 algorithm. The public key will be added to `.ssh/authorized_keys`.

6. Add the line `AuthenticationMethods publickey` to explicitly specify the authentication method.

7. Change line `UsePAM yes` to `UsePAM no`.

After all changes, perfomr the restart of the OpenSSH server process:
`sudo systemctl restart sshd`.

The resultant configuration file has the following form:

Include /etc/ssh/sshd_config.d/*.conf

Port 4596
PermitRootLogin no
PubkeyAuthentication yes
UsePAM no

#these lines were already in the file
PasswordAuthentication no
ChallengeResponseAuthentication no
X11Forwarding yes
PrintMotd no
AcceptEnv LANG LC_*
Subsystem	sftp	/usr/lib/openssh/sftp-server

In order to connect to the publicly available server through ssh use the following command:

ssh -i inno inno@18.216.251.167 -p 4596

### 3.2.3. Configure firewall
We configure firewall with use of both iptables and inbound rules in AWS console (see 3.1.6).

Configuration of firewall on 10.0.11.10:

sudo iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
sudo iptables -A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

sudo iptables -A INPUT -p tcp --dport 4596 -j ACCEPT
sudo iptables -A OUTPUT -p tcp --dport 4596 -j ACCEPT

sudo iptables -A OUTPUT -p tcp --dport 22 -j ACCEPT

sudo iptables -A INPUT -p tcp --dport 80 -j ACCEPT
sudo iptables -A OUTPUT -p tcp --dport 80 -j ACCEPT

sudo iptables -A INPUT -p tcp --dport 443 -j ACCEPT
sudo iptables -A OUTPUT -p tcp --dport 443 -j ACCEPT

sudo iptables -A INPUT -j DROP
sudo iptables -A OUTPUT -j DROP

Configuration of firewall on 10.0.12.10 and 10.0.12.10:

sudo iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
sudo iptables -A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

sudo iptables -A INPUT -p tcp --dport 22 -j ACCEPT
sudo iptables -A OUTPUT -p tcp --dport 22 -j ACCEPT

sudo iptables -A INPUT -p tcp --dport 8000 -j ACCEPT
sudo iptables -A OUTPUT -p tcp --dport 8000 -j ACCEPT

sudo iptables -A INPUT -j DROP
sudo iptables -A OUTPUT -j DROP

Configuration of firewall on 10.0.13.10:

sudo iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
sudo iptables -A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

sudo iptables -A INPUT -p tcp --dport 22 -j ACCEPT
sudo iptables -A OUTPUT -p tcp --dport 22 -j ACCEPT

sudo iptables -A INPUT -p tcp --dport 5432 -j ACCEPT
sudo iptables -A OUTPUT -p tcp --dport 5432 -j ACCEPT

sudo iptables -A INPUT -p tcp --dport 5672 -j ACCEPT
sudo iptables -A OUTPUT -p tcp --dport 5672 -j ACCEPT

sudo iptables -A INPUT -j DROP
sudo iptables -A OUTPUT -j DROP




## 3.3. Configure Nginx
Currently the server is capable only of HTTP connection that was a source of multiple bugs in frontend (see opende issues). As you can see the connection is not secure.


### 3.3.1 Generate certificate for HTTPS
Before establishing the secure connections through HTTPS, we need to generate the key and the certificate for it. With use of following command we generate the key and the certificate:

$ sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ./ssl/doctorina-api.com.key -out ./ssl/doctorinna-api.com.crt

After filling the credentials for the certificate we can see the generated certificate and the key.

*Creation of the certificate and the key for the Nginx secure connection establishing*

### 3.3.2 Configure Nginx to use SSL
After we configure connection through HTTPS with use of SSL protocol and redirect all connection through HTTP to HTTPS. To do so we change the configuration file `default.conf` for the Nginx:
*Add SSL certificate and key, moreover configure redirection from HTTP to HTTPS*

Next we update `Dockerfile`, adding the copying of the SSL certificate and key into container:

After all configuration we try to build the container and push to DockerHub

Next, to check the correctness of our configuration we run the backend part with use of Docker containers and check the connection to the web interface of the API:
Run the application on Docker containers with use of:

$ docker-compose up

As we can see, the connection is established through HTTPS:

With use of browser we can check the credentials of our certificate:

### 3.3.3 Configure load-balancing
Next our team configured load-balancing on Nginx. To do so, we change the `default.conf` Nginx configuration file by adding copy of services and choose the strategy to perform load-balancing:
*Least connection strategy was choosen and the load was distributed across 3 copies of gunicorn service*

After it we update `docker-compose.yml`. We removed explicit mapping of ports for `django_gunicorn` and introduce scaling for it to create copies of the service:
*Removing explicit mapping of ports for `django_gunicorn` and add `scale = 3` for creation of copies*

Checking the work of the Nginx by accessing the web interface of the API:

## 3.4. Automate configuration

According to the architecture of our VPC, there is one public and three private servers. This means that we cannot simply connect to all ther servers from the outer Internet (e.g. from Github Actions runner). Therefore in order to deploy the application, we connect to the public server and then spread all the configurations further to the private subnets from it. For the latter we used IT automation tool - Ansible. It allows us to check servers for needed packages and spin up the Docker containers on them.

### 3.4.1. Setup control server
As the public server is the central Ansible server (i.e. it initiates the process), it needs special configuration.

Firstly, one needs to install Python (of version 3.5 at least):

sudo apt update
sudo apt install python3.9 python3.9-distutils

Python might come without PIP, so the next step is to fix it:

curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python3.9 get-pip.py

Finally, the only left is to install ansible with PIP:

python3.9 -m pip install virtualenv
python3.9 -m virtualenv venv
. venv/bin/activate
pip install ansible

Running these commands on the server:






### 3.4.2. Playbook

The essential part of ansible configuration is *playbook.yml* file. It describes 6 steps (plays) to perform: the control node collects information (facts) from all the servers, assures that Docker is properly installed on each of them and then checks/runs the containers. Firstly, the database and broker containers are checked, if they are not currently running then they are started (there is no need to restart them every CI/CD pipeline). The last two plays similarly correspond to API and NGINX containers, however they also involve pulling of the latest images from DockerHub and restarting of the containers.

*playbook.yml:*

---

- name: Gather facts
  hosts: all
  become: yes
  gather_facts: yes

- name: Docker
  hosts: all
  become: yes
  gather_facts: no
  roles:
    - docker

- name: Database container
  hosts: db_server
  become: yes
  gather_facts: no
  roles:
    - db

- name: Broker container
  hosts: broker_server
  become: yes
  gather_facts: no
  roles:
    - broker

- name: API container
  hosts: api_servers
  become: yes
  gather_facts: no
  roles:
    - api

- name: NGINX container
  hosts: nginx_server
  become: yes
  gather_facts: no
  roles:
    - nginx

### 3.4.3. Roles

Each of the roles's actions are described in `<role>/tasks/main.yml` files

These files specify the state we want the servers to be in. Important to say that if some state described in some task is already satisfied on the server, *Ansible* will proceed further, and only otherwise it will perform corresponding actions. Let us consider the first task of *docker* role, it states that we want 3 APT packages to be installed on the server, i.e. on each run *Ansible* will check that this condition is satisfied, and if not it will do the installation (with APT cache update beforehand).

`roles/docker/tasks/main.yml`:

---

- name: Install prerequisites
  apt:
    name:
      - ca-certificates
      - curl
      - gnupg
    update_cache: yes
  
- name: Add APT key
  apt_key:
    url: https://download.docker.com/linux/ubuntu/gpg
    keyring: /usr/share/keyrings/docker-archive-keyring.gpg

- name: Add APT repository
  apt_repository:
    repo: "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu {{ ansible_distribution_release }} stable"
    filename: docker

- name: Install Docker
  apt:
    name:
      - docker-ce
      - docker-ce-cli
      - containerd.io
    update_cache: yes

- name: Start and enable Docker service
  systemd:
    name: docker
    state: started
    enabled: yes

- name: Add user permissions
  user:
    name: "{{ ansible_user }}"
    append: yes
    groups:
      - docker

- name: Install Python3
  apt:
    name: python3
    
- name: Install PIP
  apt:
    name: python3-pip

- name: Upgrade pip
  pip:
    name: pip
    state: latest
    executable: pip3

- name: Install Docker SDK for Python
  pip:
    name:
      - docker
    executable: pip3

- name: Log into DockerHub
  docker_login:
    username: "{{ REGISTRY_USER }}"
    password: "{{ REGISTRY_PASSWORD}}"

`roles/db/tasks/main.yml`:

---

- name: Run database container (if not running)
  docker_container:
    name: "{{ DB_CONTAINER }}"
    image: "{{ DB_IMAGE }}"
    env:
      POSTGRES_DB: "{{ DB_NAME }}"
      POSTGRES_USER: "{{ DB_USER }}"
      POSTGRES_PASSWORD: "{{ DB_PASSWORD }}"
    ports:
      - "{{ DB_PORT }}:5432"
    volumes:
      - "{{ DB_VOLUME }}:/var/lib/postgresql/data"

`roles/broker/tasks/main.yml`:

---

- name: Run broker container (if not running)
  docker_container:
    name: "{{ BROKER_CONTAINER }}"
    image: "{{ BROKER_IMAGE }}"
    ports:
      - "{{ BROKER_PORT }}:5672"

`roles/api/tasks/main.yml`:

---

- name: Pull API image and run container
  docker_container:
    name: "{{ API_CONTAINER }}"
    image: "{{ REGISTRY }}/{{ API_IMAGE }}"
    pull: yes
    env:
      SECRET_KEY: "{{ SECRET_KEY }}"
      DEBUG: "{{ DEBUG }}"
      POSTGRES_DB: "{{ DB_NAME }}"
      POSTGRES_USER: "{{ DB_USER }}"
      POSTGRES_PASSWORD: "{{ DB_PASSWORD }}"
      POSTGRES_HOST: "{{ DB_HOST }}"
      BROKER_HOST: "{{ BROKER_HOST }}"
      ADMIN_USERNAME: "{{ ADMIN_USERNAME }}"
      ADMIN_EMAIL: "{{ ADMIN_EMAIL }}"
      ADMIN_PASSWORD: "{{ ADMIN_PASSWORD }}"
    ports:
      - "{{ API_PORT }}:8000"
    volumes:
      - static:/static

`roles/nginx/tasks/main.yml`:

---

- name: Pull NGINX image and run container
  docker_container:
    name: "{{ NGINX_CONTAINER }}"
    image: "{{ REGISTRY }}/{{ NGINX_IMAGE }}"
    pull: yes
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - static:/static

### 3.4.4. Hosts

Information on the hosts is located in the `hosts.ini` file.

*hosts.ini*:

server1 ansible_host=localhost ansible_connection=local
server2 ansible_host=10.0.12.10
server3 ansible_host=10.0.12.11
server4 ansible_host=10.0.13.10

[nginx_server]
server1

[api_servers]
server[2:3]

[db_server]
server4

[broker_server]
server4

[local]
server1

[remote]
server[2:4]

We can see that there are 4 servers specified, also they are distributed among different groups (marked with braces). This allows us to write logical names of hosts in the playbook as well as assign same variables for all corresponding servers just once instead of repeating them for each server separately.

### 3.4.5. Variables

The variables are stored in `group_vars/local.yml`, `group_vars/remote.yml`, `vars.yml` and `roles/<role>/defaults/main.yml`.

`group_vars/local.yml` (with test data):

ansible_user:      test
ansible_sudo_pass: test

`local.yml` stores username and sudo password of public server user (is named local because from the Ansible prospective our public server is actually localhost).

`group_vars/remote.yml` (with test data):

ansible_user:             test
ansible_private_key_file: /home/test/key/id_ed25519

`remote.yml` stores username of private server users and path to ssh private key for connection from public to private servers

`vars.yml` (with test data):

REGISTRY_USER:     test
REGISTRY_PASSWORD: test
REGISTRY:          test
DB_NAME:           test
DB_USER:           test
DB_PASSWORD:       test
SECRET_KEY:        test
DEBUG:             False | quote
DB_HOST:           10.0.13.10
BROKER_HOST:       10.0.13.10
ADMIN_USERNAME:    test
ADMIN_EMAIL:       [e-mail]
ADMIN_PASSWORD:    test

The variables in this file are required to pull Docker images and run containers.

`roles/api/defaults/main.yml`:

---

API_CONTAINER: doctorinna-api
API_IMAGE: doctorinna-api

API_PORT: 8000

Similarly other roles have such `defaults/main.yml` files with insensitive data such as container and image names.

## 3.5. Set up Github Action

We have set up Github workflow on push / pull request to the master branch. It consists of 2 jobs: continous integration and deployment. The former checks the project against Python's Flake8 linter and then runs the tests. Second job builds the NGINX and API images and pushes them to DockerHub, then the ansible directory as well as some files with variables are copied to the public server and the ansible playbook is run there.
To be more precise, when the pull request is created only the first job is initated, while on the push both jobs run.
In order not to expose sensitive data we used *secrets* of Github which allow to store variables securely and access them during CI/CD pipeline.

[The link to the workflow file](https://github.com/Doctorinna/backend/blob/master/.github/workflows/ci-cd-master.yml)


## 3.6. Deploy frontend
[Here](https://github.com/Denisalik/ssad_frontend_deploy) you can check the repository which contains the source code for the frontend.
We deploy application frontend on other instance

You can see it [here](http://3.15.229.190/), beware that you need to disable your browser's TLS certificate checks, as browser doesn't trust self-signed certificates and frontend request backend https server. [This can help if you use chrome](https://medium.com/idomongodb/chrome-bypassing-ssl-certificate-check-18b35d2a19fd).
Try this if you are on linux:
1. Use this to find your chrome executable `whereis google-chrome`. If it is something like `google-chrome: /usr/bin/google-chrome /usr/share/man/man1/google-chrome.1.gz`, you don't need to change below.
2. Use this command to launch chrome without TLS checks `/usr/bin/google-chrome --ignore-certificate-errors --ignore-urlfetcher-cert-requests &> /dev/null`. If you have different path, change `/usr/bin/google-chrome` to your path.

*The application available at [3.15.229.190](http://3.15.229.190/)*

## 3.7. Load Testing
To perform load testing of our application we used [Yandex.Tank](https://yandex.com/dev/tank/) - open-source load testing tool.

### 3.7.1. Configuration

Configuration file `load.yml`:

phantom:
  address: 18.216.251.167:443
  ssl: true
  ammo_type: phantom
  ammofile: ammo.txt
  load_profile:
    load_type: rps
    schedule: line(1, 80, 3m)
  writelog: all
console:
  enabled: true
telegraf:
  enabled: false
overload:
  enabled: true
  package: yandextank.plugins.DataUploader
  token_file: "token.txt"

`phantom` - specifes that the *phantom* load generator will be used (is written in C++)
`address` - IP address of the target
`ammo_type: phantom` - the method of specifying requests via special file (see ammo.txt below)
`ammofile` - the name of the file with requests description
`schedule: line(1, 80, 3m)` - specification of the load, in our case it was linear from 1 rps to 80 rps and lasted for 3 minutes
`writelog` - option to save the logs of requests and responses
`console` - option to show information on testing in the console
`telegraf` - module intended to monitor target server's metrics such as CPU, dick, etc. (wasn't used in our case)
`overload` - enables the UI plugin with graphical dashboards
`token_file` - in order to use the overload plugin one needs to [sign up](https://overload.yandex.net/login/?next=/), get API token from the profile and paste in into the file with the name provided in this argument

### 3.7.2. Requests

`ammo.txt`:

52 get
GET /api/risks/diseases/ HTTP/1.0
User-Agent: tank

54 get
GET /api/risks/categories/ HTTP/1.0
User-Agent: tank

53 get
GET /api/risks/questions/ HTTP/1.0
User-Agent: tank

1406 post
POST /api/risks/response/ HTTP/1.0
User-Agent: tank
Content-Type: application/json
Content-Length: 1300

[{"question": "1", "answer": 19}, {"question": "2", "answer": "Male"}, {"question": "3", "answer": 80}, {"question": "4", "answer": 185}, {"question": "5", "answer": "No"}, {"question": "6", "answer": "No"}, {"question": "7", "answer": "No"}, {"question": "8", "answer": "No"}, {"question": "9", "answer": "No"}, {"question": "10", "answer": "No"}, {"question": "11", "answer": "No"}, {"question": "12", "answer": "No"}, {"question": "13", "answer": "No"}, {"question": "14", "answer": "No"}, {"question": "15", "answer": "No"}, {"question": "16", "answer": "No"}, {"question": "17", "answer": "No"}, {"question": "18", "answer": "No"}, {"question": "19", "answer": 110}, {"question": "20", "answer": 90}, {"question": "21", "answer": "No"}, {"question": "22", "answer": "No"}, {"question": "23", "answer": "Normal"}, {"question": "24", "answer": "Normal"}, {"question": "25", "answer": "No"}, {"question": "26", "answer": "Never smoked"}, {"question": "27", "answer": "No"}, {"question": "28", "answer": "No"}, {"question": "29", "answer": "No"}, {"question": "30", "answer": "Never worked"}, {"question": "31", "answer": "I don't know"}, {"question": "32", "answer": "Compute from my height and weight"}, {"question": "33", "answer": "Urban"}, {"question": "34", "answer": "Tatarstan Republic of"}]

The load tester will "shoot" with 4 requests. We considered important to use the last POST request because "under the hood" it runs the machine learning model which might cause more load on CPU in comparison to other handlers.

### 3.7.3. Run Test

We ran testing in the Docker container:
`docker run -ti --name tank -v $(pwd):/var/loadtest/ --network host direvius/yandex-tank`
(files ammo.txt and token.txt should be in the current directory)

Since we enabled console in the configuration, we have such an output with percentiles of response time, HTTP and Net codes of the reponses, some other metrics, current RPS and link to the analytic service with different plots (due to enabled overload plugin)

### 3.7.4. Discussion

We ran testing on 2 application configurations: with 1 and 2 API instances

- 1 instance:

Result: https://overload.yandex.net/488889

Response time quantiles:

Net codes:

- 2 instances:

Result: https://overload.yandex.net/488863

Response time quantiles:

Net codes:

**Comparison conclusion:**
In the first case one can see that the timeouts started from **40 rps** while in the second from **64 rps** (110 net code - ETIMEDOUT Connection timed out). Therefore, the application with 2 API instances can sustain more load

## 3.8. Demo
For demonstration 2 videos were recorded. [First one](https://youtu.be/OJd06Psdnh0) shows the architecture of application in terms of EC2 instances on AWS platform. Moreover, it shows the API's work and Continuous Integration & Continous Delivery set up in Github Actions. Also, the load testing with use of YandexTank was shown during the video.
[Second video](https://youtu.be/CQO2Jz2g5-M) shows the deployed application work from [here](http://3.15.229.190/)

## 4. Troubleshooting
This section presents the difficulties we faced during the project deployment.

### 4.1. Hanged SSH connection
While we configured iptables, we blocked ourselves with `sudo iptables -A INPUT -j DROP`(ssh connection just hanged, writing anything didn't display in terminal). Problem was that we didn't write command for accepting established connections. Restart of the instance solved issue and dropped iptables successfully and we procede to write commands written in `Improve securite of instance with public IP address` section.

### 4.2. Problem of self-signed certificate
Chrome browser checks for TLS certificates and didn't display frontend page correctly(content wasn't rendered, component responsible for it was just blank).

Requests just dropped on client-side, because chrome didn't trust self-signed certificates. Executing chrome with special arguments(`usr/bin/google-chrome --ignore-certificate-errors --ignore-urlfetcher-cert-requests &> /dev/null`) solved issue for testing purposes. But for production, buying certificates is an option to solve this problem.
## 5. Conclusion

During the project we achieve main goal to set up the continuous integration and deployment (CI/CD pipeline) for the Doctorinna API in AWS infrastructure. Our team performed all 4 tasks needed for the success. We set up the AWS infrastructure, configured HTTPS connection and load-balancing in Nginx, dockerized services inside project, and created Github actions for continous intergration & continous delivery. Despite all difficulties we faced, we can claim that our project was overall success and we managed to deploy our application.