# Keeping peace in Peaceland

Vincent Courty
Luca Moorghen
Raphael Tepe
Nathan Habib

## Context and Client Demands

Peaceland is a blessed country, led by an affable and clear-sighted
ruler. He takes great pride in its effort to bring peace, happiness,
and harmony to all its citizens. 

To do so, they heavily rely on their peacemakers. A governmental
agency dedicated to make peace around the country. To reach their
ambition, they bring assistance to any agitated person and help them
to recover peace. More generally they help citizen to stay in line
with their country harmonious goal

To help its peacemakers squads, Peaceland engineers have created a
working autonomous drone called peacewatcher.

They need you to create the program that will receive and manage
peacewatchers’s data.
This program must:

* Store every peacewatcher data
* Trigger alerts
* Enable peacemaker officers to perform analysis on peacewatcher data

### Description of the drones

Each peacewatcher sends a report every minute.
One report contains:
* Peacewatcher id
* Peacewatcher current location (latitude, longitude)
* Name of surrounding  citizen (identify with facial recognition) with their computed «peacescore». Peacescore being recomputed for every report.
* Words heard by the peacewatcher in its surrounding

### Alerts
When a citizen peacescore is bad, your program must trigger an alert
with the location of the peacewatcher and the name of the agitated citizen.
Peacemakers will take it from there and help the person to find peace.

They may send him to a peacecamp. In such camp citizen learn to reach
happiness following the ideas of the beneveland leader of Peaceland.
Or they will put him in a sustainable and never ending peace state.
This alert must be triggered as quickly as possible because an agitated
citizen may spread its lack of peace to other citizens. Thus, the
peacemaker reaction must be as fast as possible.

### Statistics
Peacemakers are convinced that we need to keep every peacewatcher
report in order to make statistics and improve their Peaceland harmony.
But they still don’t know what kind of question/statistic they will
want to address.
Peaceland engineer estimate that when the first wave of peacewatcher
will be operational the sum of all their daily report will weight 200Gb

They also estimate that less than 1% of peacewatcher report contains alert.

### Failed attempts
To create a POC of the program, Peaceland hired a team of data-scientists and 
despite all their efforts, this team have not been able to set up a scalable
program that can handle the load.

## Project

Peaceland understands this is beyond their team limits, it can not put in place a programm to deal with the drone’s data. Peaceland asks you for advice to design an architecture allowing them to create a product they could sell to different police forces.

It's up to you to report and recommend the right architecture.

## Architecture

graph LR
subgraph Peacewatchers
P1(( ))
P2(( ))
P3(( ))
end
P1 --> ST
P2 --> ST
P3 --> ST
ST[[Stream]]
ST2[[Stream]]
subgraph Treatment
SC1[Kafka-Connect]
SC2[Stream Consumer Alert]
end
ST --> SC1
ST --> SC2
DL[(Amazon S3)]:::data
PM{Peacemakers}
SC1 --> DL
SC2 --> ST2 --> PM
DS[Data-Scientists]
DL --> DS

classDef data fill:#3498DB,stroke:BLACK,stroke-width:2px 

# Questions

### What technical/business constraints should the data storage component of the program architecture meet to fulfill the requirement described by the customer in paragraph «Statistics» ?

* Need fast writing
* Allow document compression
* Need to be separated from the processing.
* Scalable (horizontal scaling)
* Need to be:
* Consistent for distributed computing
* Partition tolerant, need to be up and running even when a node fails

The is actually a big data problem because of the huge amount of data we have to analyze and store. The use of a stream is a good choice to handle scalability and parallelism with high volumes of data.

Peacemakers also want to keep all the data although they don’t know exactly which information they need to make their questions/statistics, therefore the best approach would be to use a data lake to store them without relationality.

### What business constraint should the architecture meet to fulfill the requirement describe in the paragraph «Alert»? Which component to choose?

* Needs to run if a peacewatcher fails
* Handle high volume of data every minute and almost no data between every minute
* We need to still be able to send alerts even when the database is down.
* Be able to add functionalities without changing the architecture and no downgrade in performance.
* The architecture should detect rapidly bad peacescores in order to alert peacemakers who will send the citizens in a peacecamp. -> Near real time processing

With all those constraints we choose to use a stream and stream reader components.

### What mistake(s) from Peaceland can explain the failed attempt?

Peaceland only hired a team of data-scientists, their skills allow them to efficiently work on data but not to design an architecture. Scalability issues are surely the result of bad architecture unable to handle a **large data flow**, and too **many points of failure**.

### Peaceland has likely forgotten some technical information in the report sent by the drone. In the future, this information could help Peaceland make its peacewatchers much more efficient. Which information?

The addition of timestamps would be useful to keep a traceability of reports and monitoring the evolution of peacescores for each citizen. With this we could for example prioritize the new reports in which the citizen had a peacescore near the threshold which would send him in peacecamps. It is important information as well for the team of data-scientists who will be able to make statistics and adapt their strategy.