# ELK-Stack Tutorial
The ELK Stack is a powerful open-source platform composed of **Elasticsearch**, **Logstash**, and **Kibana** that enables users to ingest, process, store, search, and visualize large volumes of structured and unstructured data in real time.
The ELK Stack excels at collecting time‑series data from distributed devices (such as the Raspberry Pi Pico W and the ESP32) and sensors, turning these into actionable insights. Its scalability allows efficient storage and rapid querying of vast IoT datasets, while its flexible ingestion pipeline (Logstash/Beats) and real‑time dashboards (Kibana) enable monitoring, anomaly detection, and predictive maintenance— all essential for managing complex IoT deployments across many devices.
# Part 1: Setting up the ELK-Stack
We will follow the following tutorial for setting up the ELK-Stack:
https://github.com/deviantony/docker-elk
**Requirements:** You will first need to install [Docker](https://docs.docker.com/engine/install/).
## 🧠 ELK Stack Tutorial with Docker (Based on deviantony/docker‑elk)
This tutorial, based on the `deviantony/docker-elk (GitHub repo)` guides you through running the ELK stack (Elasticsearch, Logstash, Kibana) on your local machine using Docker and Docker Compose.
---
## ⚙️ Requirements
- Docker Engine ≥ 18.06
- Docker Compose ≥ 2.0 (Docker Compose CLI)
- ≥ 1.5 GB RAM available to Docker
- On **Linux**, user must have Docker daemon access
---
## 📦 Setup & Bring Up the Stack
1. **Clone the repo**
```bash
git clone https://github.com/deviantony/docker-elk.git
cd docker-elk
```
2. **Initial setup**
```bash
docker compose up setup
```
This sets up default users (`elastic`, `logstash_internal`, `kibana_system`) with passwords from `.env`.
3. **Start the stack**
```bash
docker compose up
```
Or use `-d` to detach and run in the background.
4. **Access Kibana**
Wait \~1 minute for Kibana to initialize, then open:
```
http://localhost:5601
```
**Username:** `elastic`
**Password:** `changeme` (unless changed in `.env`).
**Please change the password immediately after accessing Kibana. This can be done in the UI, and more simply with the .env file.**
---
## 🔐 User Authentication
For increased security, reset passwords from defaults:
```bash
docker compose exec elasticsearch \
bin/elasticsearch-reset-password --batch --user elastic
docker compose exec elasticsearch \
bin/elasticsearch-reset-password --batch --user logstash_internal
docker compose exec elasticsearch \
bin/elasticsearch-reset-password --batch --user kibana_system
```
Update `.env` with new passwords, then restart relevant services:
```bash
docker compose up -d logstash kibana
```
---
## 🗄️ Data Injection
* **Netcat (TCP input)**:
```bash
cat /path/to/logfile.log | nc --send-only localhost 50000
```
* Or **upload sample data** via Kibana UI.
---
## 🧹 Cleanup & Reset
* **Stop but keep data**:
```bash
docker compose down
```
* **Remove everything (volumes included)**:
```bash
docker compose down -v
```
---
## 🛠️ Version Management
* Main branch typically tracks Elastic 9.x
* To use another version, modify `.env`:
```env
ELASTIC_VERSION=<version>
```
* Rebuild and re-run:
```bash
docker compose build
docker compose up setup
docker compose up
```
---
## ⚙️ Configuration & Tuning
* **Ports exposed**:
* Elasticsearch: 9200 (HTTP), 9300 (transport)
* Logstash: 5044 (Beats), 50000 (TCP), 9600 (monitor)
* Kibana: 5601
* **JVM tuning** via environment variables:
```yaml
elasticsearch:
environment:
ES_JAVA_OPTS: "-Xms512m -Xmx512m"
logstash:
environment:
LS_JAVA_OPTS: "-Xms256m -Xmx256m"
```
Adjust based on available memory.
* **Custom configuration**:
* `elasticsearch/config/elasticsearch.yml`
* `logstash/config/logstash.yml` + pipelines in `logstash/pipeline/`
* `kibana/config/kibana.yml`
---
## ➕ Extensibility
* Add plugins: modify Dockerfiles under `elasticsearch/`, `logstash/`, or `kibana/`.
* Use extensions in `extensions/` (e.g. TLS, Fleet).
* Scale Elasticsearch cluster via Docker Compose scaling.
* Enable Beats, Curator, Fleet, and more.
---
## 🧾 Summary Workflow
1. Clone the project
2. `docker compose up setup`
3. `docker compose up -d`
4. Open Kibana at `http://localhost:5601`
5. Load data via Logstash or Kibana
6. Explore, visualize, and refine your setup
---
## 🔍 Going Further
* Dive deeper into each component’s config files
* Explore Beats (Filebeat, Metricbeat, etc.)
* Harden your stack for production
* Scale and monitor in distributed environments
---
# Part 2: Sending Sensor Data To ELK-Stack
## 📡 Raspberry Pi Pico W → ELK Stack with MicroPython
We’ll simulate sensor readings (random values) on the Pico W and forward them as HTTP POST requests to Logstash’s TCP input, which writes into Elasticsearch.
### 🧩 1. Flash MicroPython & Install `urequests`
Ensure your Pico W is running MicroPython and you have Thonny or a similar IDE set up.
"urequests" is typically bundled. If not:
```python
try:
import urequests
except ImportError:
import upip
upip.install('micropython-urequests')
import urequests
```
---
### 🌐 2. Connect to Wi‑Fi
```python
import network, time
SSID = "YOUR_SSID"
PASSWORD = "YOUR_PASSWORD"
wlan = network.WLAN(network.STA_IF)
wlan.active(True)
wlan.connect(SSID, PASSWORD)
for _ in range(10):
if wlan.isconnected():
break
print("Connecting...")
time.sleep(1)
print("Connected:", wlan.ifconfig())
```
---
### 📨 3. Send Random Sensor Data
Here's a sample script that posts JSON to Logstash (set in your `docker-elk` `logstash/pipeline/`):
```python
import urequests, time, random, json
ENDPOINT = "http://<YOUR_HOST_IP>:50000/" # Logstash TCP input via HTTP plugin
def send_reading():
val = random.uniform(0, 100)
payload = {"sensor": "pico", "value": val, "ts": time.time()}
try:
resp = urequests.post(ENDPOINT, data=json.dumps(payload),
headers={'Content-Type':'application/json'})
resp.close()
print("Sent:", payload)
except Exception as e:
print("Error:", e)
while True:
send_reading()
time.sleep(5)
```
📋 **Important:**
* Replace `<YOUR_HOST_IP>` with your machine’s IP address on the Pico's Wi‑Fi network.
* Ensure Logstash’s pipeline configuration includes an HTTP input and forwards to Elasticsearch.
---
### 🔁 4. Configure Logstash HTTP Input
In `logstash/pipeline/http.conf`:
```conf
input {
http {
port => 50000
}
}
filter {
json { source => "message" }
}
output {
elasticsearch {
hosts => ["elasticsearch:9200"]
index => "pico-%{+YYYY.MM.dd}"
}
stdout { codec => rubydebug }
}
```
This makes Logstash listen on port 50000 for JSON payloads, parses them, and indexes into Elasticsearch.
---
### 📊 5. Visualize in Kibana
1. In Kibana, create an index pattern like `pico-*`
2. Open Discover to see posts from your Pico
3. Build visualizations: e.g., value over time line charts
---
### 🧪 6. Recap Workflow
1. Pico connects via Wi-Fi
2. Generates random values
3. Makes HTTP POST to Logstash
4. Logstash parses JSON → forwards to Elasticsearch
5. Kibana visualizes the data
---
### ✅ Notes & Tips
* You should send real sensor data from the microcontroller.
* For production, consider using Elasticsearch’s HTTP API or Python client directly, bypassing Logstash.
* Securing communications (HTTPS, authentication) on Pico is advanced—HTTPS POSTs on Pico W may need custom handling.
Sign in with Wallet
Connect another wallet