# Review ELK --- ## Komponenten - Beat (Filebeat) - **E**lasticsearch - **L**ogstash - **K**ibana --- ## Elastic - 2000: Hobby-Projekt Suchmaschine für Rezepte - 2018: Börsendotiert mit 230 MAs - 2020: Umsatz $428 Mio, >1'000 MAs - 2021: Apache2 Lizenz --> SSPL (non OSS) --- ## Overview  --- ## Beat ### "Data Shipper" --- ## Inputs - File (logfile) - Journal (systemd's journald) - Heartbeat (http, tcp, icmp) - dockbeat - githubbeat - earthquakebeat - ... --- ## Filebeat ### "tail -f" als Shipper ``` filebeat.prospectors: - type: log enabled: true paths: - /var/lib/mesos/slaves/**/stdout - /var/lib/mesos/slaves/**/stderr output.logstash: hosts: ["<myhost.example.com:5044"] index: mesos_cluster ``` --- ## Logstash ### "Data Transform Pipeline"  --- ## Outputs - Elasticsearch - S3 - Kafka - Nagios - InfluxDB - Pipe - http - ... --- ## Config ``` input { beats { host => "myhost.example.com" port => 5044 } } filter { grok { match => { 'message' => [ "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} *\[%{DATA:thread}\] *\[%{DATA:class}\] *%{DATA:method} - \<\<METHOD-END\>\>\.search\(\) customer\=\[%{DATA:CUSTOMER}\],time\(ms\)\=\[%{INT:LATENCY:int}\],recordsCount\=\[%{INT:RECORDS_COUNT:int}\],recordsTotal\=\[%{INT:RECORDS_TOTAL:int}\],entriesCount\=\[%{INT:ENTRIES_COUNT:int}\],entriesTotal\=\[%{INT:ENTRIES_TOTAL:int}\]", "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} *\[%{DATA:thread}\] *\[%{DATA:class}\] *%{DATA:method} - %{DATA:message}" ] } } } output { elasticsearch { hosts => "elastic.example.com:9200" } } ``` --- ## Grok Demo --- ## Elasticsearch - Lucene basierte Volltextsuche und NoSQL Dokumentenspeicher (JSON) - REST API  --- ## Query ``` curl localhost:9200/_search?pretty -H 'Content-Type: application/json' -d '{ "query": { "match": { "level": { "query": "INFO" } } } }' ``` --- ## Kibana ### Visualize and analyze  ### Demo