# ELK
## Groepsleden
Pieter Fiers
Dieter Draelants
Joeri Sprengers
## Common
All components require the Elasticsearch GPG key. Add it to apt:
```shell=
curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
```
The Elasticsearch repo requires HTTPS:
```shell=
sudo apt-get install apt-transport-https
```
Add the Elasticsearch apt repo to /etc/apt/sources.list.d/elastic-7.x.list:
```shell=
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
```
Update apt:
```shell=
sudo apt update
```
## Elasticsearch
Install Elasticsearch:
```shell=
sudo apt install elasticsearch
```
First step for the basic configuration of elastic is in the elasticsearch.yml:
```shell=
sudo nano /etc/elasticsearch/elasticsearch.yml
```
Here you gone gone give your cluster a name but also name your node. After this you also need to configure your hostname. After you have configured your hostname, you also specify which discovery type you gone chose, we chose single-node because we are only using 1 elastic node.
```yaml=
network.host: 193.191.177.158
cluster.name: ELK
node.name: elastic
discovery.type: single-node
```
After you have configured your elasticsearch.yml file you start up the service. The second command is if you want it to start-up at boot.
```shell=
sudo systemctl start elasticsearch
sudo systemctl enable elasticsearch
```
If you have configured everything correctly, it should work.
Now you can reach the application with localhost:9200, but you wanne reach it wit a dns you still need to configure couple of things.
First go to your dns settings, in our case its in:
```shell=
nano /etc/bind/zones/db.joeri-sprengers.sb.uclllabs.be
```
Here you add an cname record for you application.
```
elasticsearch IN CNAME @
```
After you have done this you need to restart your dns service.
```shell=
systemctl restart bind9.service
```
Now we gone create in apache a vhost, you can configure this in:
```shell=
nano /etc/apache2/sites-available/001-wwwconfig-le-ssl.conf
```
Specifically we are gone create an secure connection, so first we add the vhost in the file:
```
<VirtualHost *:443>
ServerName elasticsearch.joeri-sprengers.sb.uclllabs.be
ServerAdmin webmaster@localhost
ErrorLog ${APACHE_LOG_DIR}/elastic-error.log
CustomLog ${APACHE_LOG_DIR}/elastic-access.log combined
ProxyPass "/" "http://localhost:9200/"
Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/secure.joeri-sprengers.sb.uclllabs.be-0001/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/secure.joeri-sprengers.sb.uclllabs.be-0001/privkey.pem
</VirtualHost>
```
After you have added it to the config file you need to reload the apache service.
```shell=
systemctl reload apache2.service
```
Now we can acces the application with the link http://elasticsearch.joeri-sprengers.sb.uclllabs.be/, but it still need an certificate before it can be called a safe site. So we run certbot or wait till it is runned by the crontab:
```shell=
certbot -n --agree-tos --expand --apache --no-redirect -d secure.joeri-sprengers.sb.uclllabs.be -d supersecure.joeri-sprengers.sb.uclllabs.be -d elasticsearch.joeri-sprengers.sb.uclllabs.be
```
Now you have configured everything, and can you start configure the next part.
## Logstash
Install Logstash:
```shell=
sudo apt install logstash
```
We also need to start and enable start on reboot:
```shell=
systemctl start logstash
systemctl enable logstash
```
So actually first we tried making it work without Logstash, all went great. We could see logs get uptime feed of configured sites and such. But after enabling, getting it enabled it worked but after a few hours there were these errors saying the disk capacity was full. After pulling out our hairs for certain amount of time we found a solution that nobody mentioned in their solutions. It was under **Index Lifecycle Policies** where we found that you could cap how much data was being stored. (not comfirmed that this is actually working)
So now for logstash after the install you really only need to edit 1 file (excluding the different beats) the `pipelines.yml`. Its actually best practice to put your pipeline(s) in the `/conf.d` subdirectory, `pipelines.yml` loads every file in this directory ending with *.conf*
So this is our created pipeline. The first part is the input for which I configured 2 static file streams (for testing purposes) and one filestream that collects everything beat-related. Currently only the filebeat is linked to logstash the others are directly to elasticsearch. As you can see there is a type defined. This is used in the output part as we will see now.
```
# Beats -> Logstash -> Elasticsearch pipeline.
input {
file {
path => "/var/log/auth.log"
type => "syslog"
}
file {
path => "/var/log/apache2/access.log"
type => "apache"
}
beats {
port => "5044"
type => "beats"
}
}
```
Now for the output we have a basic system out. And then a 'if statement' that checks for the type - earlier defined in the input streams - when it matches it excecutes that specific part. They all neet to redirect to the host, beeing the elasticsearchserver and its corresponding port beeing 9200 by default. We can also give it a custom index that we can use to easly identify the dataset in kibana. For this there is also a standard beeing the name of the fileset followed by a '-' and then '%{+YYYY.MM.dd}' the last thing will automatically create the date of the dataset.
```
output {
stdout {codec => rubydebug}
if [type] == "syslog" {
elasticsearch {
hosts => ["http://elasticsearch.joeri-sprengers.sb.uclllabs.be:9200"]
index => "syslogOnly-%{+YYYY.MM.dd}"
#user => "elastic"
#password => "changeme"
}
}
if [type] == "apache" {
elasticsearch {
hosts => ["http://elasticsearch.joeri-sprengers.sb.uclllabs.be:9200"]
index => "apache_accessOnly-%{+YYYY.MM.dd}"
#user => "elastic"
#password => "changeme"
}
}
if [type] == "beats" {
elasticsearch {
hosts => ["http://elasticsearch.joeri-sprengers.sb.uclllabs.be:9200"]
index => "beatsby-%{+YYYY.MM.dd}"
#user => "elastic"
#password => "changeme"
}
}
}
```
After everychange its also important to restart logstash by writing:
```shell=
systemctl restart logstash
```
## Beats
So this part is about the different beats that we've installed (or tried to). Most of these have a similar installation and configuration but we'll address them quickly in each part.
### Filebeat
After you start Filebeat, open the Logs UI and watch your files being tailed right in Kibana. Use the search bar to filter by service, app, host, datacenter, or other criteria to track down curious behavior across your aggregated logs.
Filebeat ships with modules for observability and security data sources that simplify the collection, parsing, and visualization of common log formats down to a single command. They achieve this by combining automatic default paths based on your operating system, with Elasticsearch Ingest Node pipeline definitions, and with Kibana dashboards. Plus, a few Filebeat modules ship with preconfigured machine learning jobs.
Lastly a interesting fact. Filebeat uses a backpressure-sensitive protocol when sending data to Logstash or Elasticsearch to account for higher volumes of data. If Logstash is busy crunching data, it lets Filebeat know to slow down its read. Once the congestion is resolved, Filebeat will build back up to its original pace and keep on sending data'.
Install filebeat:
```shell=
sudo apt-get install filebeat
```
### Metricbeat
Metricbeat can be deployed on all your Linux, Windows, and Mac hosts, connect it to Elasticsearch and voila: you get system-level CPU usage, memory, file system, disk IO, and network IO statistics, and thons off other system statistics.
We had some issues with visualising some of this data though.
Install metricbeat:
```shell=
sudo apt-get install metricbeat
```
### Heartbeat
Heartbeat makes it easy to generate uptime and response time data. You can track availability of your websites, services and even see when your TLS is going to expire
Install heartbeat:
```shell=
sudo apt-get install heartbeat
```
### Auditbeat
Monitor user activity and processes, and analyze your event data in the Elastic Stack without touching auditd. Auditbeat communicates directly with the Linux audit framework, collects the same data as auditd, and sends the events to the Elastic Stack in real time.
Install auditbeat:
```shell=
sudo apt-get install auditbeat
```
This is the one that didn't work and I think it has something to do about us running on containers or something. It just always gave the error that I didn't have enough permissions while I gave the user root permissions.
## Kibana
Install Kibana:
```shell=
sudo apt install kibana
```
Add reference to Elasticsearch node:
in `/etc/kibana/kibana.yml`
```
elasticsearch.hosts: ["http://elasticsearch.joeri-sprengers.sb.uclllabs.be:9200"]
```
Configure HTTP reverse proxy:
I use a Ansible playbook to configure Nginx and run certbot - [see GitHub](https://github.com/ubipo/syb/blob/master/etc/ansible/deploy-http-server.yml#L147)
### Visualisations
There are two types classes visualisations: _Lens_ and traditional. Lens visualisations are easier to create but are less versatile (no customisation of legend, no custom aggregations...).
The easiest way to create a Lens visualisation is to use the "Discover" tool (under Kibana in the hamburger menu).
Use the "Search field names" box to find a metric to visualise. The hash indicates a number field, "t" text and so on. Click "Visualize" to open the Lens editor.
<br>
The simplest way to create a traditional visualization is to first create a dashboard under `Kibana > Dashboard` and to then click "Create new" in edit mode.
The configuration for our network usage graph looks like this:
Combined with two gauges we created a simple monitoring dash:
### Index Lifecycle Policies
We experienced quite a bit of problems involving storage capacity and partially broke the entire stack multiple times while trying to clear space on disk. Luckily there exists a better solution than manually deleting indices: "Index Lifecycle Policies".
These policies define how long logs are retained for a certain index. We added a 5GB, 4 hours retention policy to all our indices.
Google Drive
Gist
Clipboard
Sign in with Wallet
