ELK-Stack Tutorial

The ELK Stack is a powerful open-source platform composed of Elasticsearch, Logstash, and Kibana that enables users to ingest, process, store, search, and visualize large volumes of structured and unstructured data in real time.

The ELK Stack excels at collecting time‑series data from distributed devices (such as the Raspberry Pi Pico W and the ESP32) and sensors, turning these into actionable insights. Its scalability allows efficient storage and rapid querying of vast IoT datasets, while its flexible ingestion pipeline (Logstash/Beats) and real‑time dashboards (Kibana) enable monitoring, anomaly detection, and predictive maintenance— all essential for managing complex IoT deployments across many devices.

Part 1: Setting up the ELK-Stack

We will follow the following tutorial for setting up the ELK-Stack: https://github.com/deviantony/docker-elk

Requirements: You will first need to install Docker.

🧠 ELK Stack Tutorial with Docker (Based on deviantony/docker‑elk)

This tutorial, based on the deviantony/docker-elk (GitHub repo) guides you through running the ELK stack (Elasticsearch, Logstash, Kibana) on your local machine using Docker and Docker Compose.

⚙️ Requirements

Docker Engine ≥ 18.06
Docker Compose ≥ 2.0 (Docker Compose CLI)
≥ 1.5 GB RAM available to Docker
On Linux, user must have Docker daemon access

📦 Setup & Bring Up the Stack

Clone the repo

git clone https://github.com/deviantony/docker-elk.git
cd docker-elk

Initial setup
```
docker compose up setup
```
This sets up default users (elastic, logstash_internal, kibana_system) with passwords from .env.
Start the stack
```
docker compose up
```
Or use -d to detach and run in the background.
Access Kibana Wait ~1 minute for Kibana to initialize, then open:
```
http://localhost:5601
```
Username: elastic Password: changeme (unless changed in .env).

Please change the password immediately after accessing Kibana. This can be done in the UI, and more simply with the .env file.

🔐 User Authentication

For increased security, reset passwords from defaults:

docker compose exec elasticsearch \
  bin/elasticsearch-reset-password --batch --user elastic

docker compose exec elasticsearch \
  bin/elasticsearch-reset-password --batch --user logstash_internal

docker compose exec elasticsearch \
  bin/elasticsearch-reset-password --batch --user kibana_system

Update .env with new passwords, then restart relevant services:

docker compose up -d logstash kibana

🗄️ Data Injection

Netcat (TCP input):

cat /path/to/logfile.log | nc --send-only localhost 50000

Or upload sample data via Kibana UI.

🧹 Cleanup & Reset

Stop but keep data:
```
docker compose down
```
Remove everything (volumes included):
```
docker compose down -v
```

🛠️ Version Management

Main branch typically tracks Elastic 9.x
To use another version, modify .env:
```
ELASTIC_VERSION=<version>
```

Rebuild and re-run:

docker compose build
docker compose up setup
docker compose up

⚙️ Configuration & Tuning

Ports exposed:
- Elasticsearch: 9200 (HTTP), 9300 (transport)
- Logstash: 5044 (Beats), 50000 (TCP), 9600 (monitor)
- Kibana: 5601

JVM tuning via environment variables:

elasticsearch:
  environment:
    ES_JAVA_OPTS: "-Xms512m -Xmx512m"
logstash:
  environment:
    LS_JAVA_OPTS: "-Xms256m -Xmx256m"

Adjust based on available memory.

Custom configuration:
- elasticsearch/config/elasticsearch.yml
- logstash/config/logstash.yml + pipelines in logstash/pipeline/
- kibana/config/kibana.yml

➕ Extensibility

Add plugins: modify Dockerfiles under elasticsearch/, logstash/, or kibana/.
Use extensions in extensions/ (e.g. TLS, Fleet).
Scale Elasticsearch cluster via Docker Compose scaling.
Enable Beats, Curator, Fleet, and more.

🧾 Summary Workflow

Clone the project
docker compose up setup
docker compose up -d
Open Kibana at http://localhost:5601
Load data via Logstash or Kibana
Explore, visualize, and refine your setup

🔍 Going Further

Dive deeper into each component’s config files
Explore Beats (Filebeat, Metricbeat, etc.)
Harden your stack for production
Scale and monitor in distributed environments

Part 2: Sending Sensor Data To ELK-Stack

📡 Raspberry Pi Pico W → ELK Stack with MicroPython

We’ll simulate sensor readings (random values) on the Pico W and forward them as HTTP POST requests to Logstash’s TCP input, which writes into Elasticsearch.

🧩 1. Flash MicroPython & Install `urequests`

Ensure your Pico W is running MicroPython and you have Thonny or a similar IDE set up.

"urequests" is typically bundled. If not:

try:
    import urequests
except ImportError:
    import upip
    upip.install('micropython-urequests')
    import urequests

🌐 2. Connect to Wi‑Fi

import network, time

SSID = "YOUR_SSID"
PASSWORD = "YOUR_PASSWORD"

wlan = network.WLAN(network.STA_IF)
wlan.active(True)
wlan.connect(SSID, PASSWORD)
for _ in range(10):
    if wlan.isconnected():
        break
    print("Connecting...")
    time.sleep(1)
print("Connected:", wlan.ifconfig())

📨 3. Send Random Sensor Data

Here's a sample script that posts JSON to Logstash (set in your docker-elk logstash/pipeline/):

import urequests, time, random, json

ENDPOINT = "http://<YOUR_HOST_IP>:50000/"  # Logstash TCP input via HTTP plugin

def send_reading():
    val = random.uniform(0, 100)
    payload = {"sensor": "pico", "value": val, "ts": time.time()}
    try:
        resp = urequests.post(ENDPOINT, data=json.dumps(payload),
                              headers={'Content-Type':'application/json'})
        resp.close()
        print("Sent:", payload)
    except Exception as e:
        print("Error:", e)

while True:
    send_reading()
    time.sleep(5)

📋 Important:

Replace <YOUR_HOST_IP> with your machine’s IP address on the Pico's Wi‑Fi network.
Ensure Logstash’s pipeline configuration includes an HTTP input and forwards to Elasticsearch.

🔁 4. Configure Logstash HTTP Input

In logstash/pipeline/http.conf:

input {
  http {
    port => 50000
  }
}

filter {
  json { source => "message" }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "pico-%{+YYYY.MM.dd}"
  }
  stdout { codec => rubydebug }
}

This makes Logstash listen on port 50000 for JSON payloads, parses them, and indexes into Elasticsearch.

📊 5. Visualize in Kibana

In Kibana, create an index pattern like pico-*
Open Discover to see posts from your Pico
Build visualizations: e.g., value over time line charts

🧪 6. Recap Workflow

Pico connects via Wi-Fi
Generates random values
Makes HTTP POST to Logstash
Logstash parses JSON → forwards to Elasticsearch
Kibana visualizes the data

✅ Notes & Tips

You should send real sensor data from the microcontroller.
For production, consider using Elasticsearch’s HTTP API or Python client directly, bypassing Logstash.
Securing communications (HTTPS, authentication) on Pico is advanced—HTTPS POSTs on Pico W may need custom handling.