:::warning ELK 安裝 新增一個ELK的資料夾並移動到ELK mkdir -p /home/sean/ELK cd /home/sean/ELK Elasticsearch 安裝 下載Elasticsearch sudo su wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.5.3-amd64.deb 安裝Elasticsearch dpkg -i elasticsearch-8.5.3-amd64.deb systemctl start elasticsearch 驗證Elasticsearch https://自己的ip:9200/ 重置密碼(elastic和kibana_system) /usr/share/elasticsearch/bin/elasticsearch-reset-password -i -u elastic /usr/share/elasticsearch/bin/elasticsearch-reset-password -i -u kibana_system Kibana 安裝 下載Kibana wget https://artifacts.elastic.co/downloads/kibana/kibana-8.5.3-amd64.deb 安裝Kibana dpkg -i kibana-8.5.3-amd64.deb Kibana 設定 vim /etc/kibana/kibana.yml 最上方加入 server.host: "0.0.0.0" 編輯完畢後，啟用 Kibana systemctl start kibana 測試Kibana http://xxx.xxx.xxx.xxx:5601/ 取得token /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana 取得驗證碼 /usr/share/kibana/bin/kibana-verification-code Logstash 下載 wget https://artifacts.elastic.co/downloads/logstash/logstash-8.5.3-amd64.deb Logstash 安裝 dpkg -i logstash-8.5.3-amd64.deb ::: -------------------------------------------------------- :::info :::success 匯入資料 ![](https://hackmd.io/_uploads/rksLFj8_2.png) ![](https://hackmd.io/_uploads/S1O9FiLdn.png) 重設帳號實有先設定帳號名稱為elastic /usr/share/elasticsearch/bin/elasticsearch-reset-password -i -u elastic ![](https://hackmd.io/_uploads/S1EfqiL_3.png) ![](https://hackmd.io/_uploads/H17j5oIun.png) ![](https://hackmd.io/_uploads/ry7HsjLdh.png) ![](https://hackmd.io/_uploads/B1dFsjUd3.png) ![](https://hackmd.io/_uploads/HJrm3sIu2.png) ![](https://hackmd.io/_uploads/HJPCjoI_n.png) Mapping更改 {"properties": { "@timestamp": {"type": "date"}, "audience_count": {"type": "long"}, "audience_fresh_critics_count": {"type": "long"}, "audience_rating": {"type": "long"}, "audience_rotten_critics_count": {"type": "long"}, "audience_status": {"type": "text","fields": {"keyword": {"type": "keyword","ignore_above": 256}}}, "audience_top_critics_count": {"type": "long"}, "cast": {"type": "text","fields": {"keyword": {"type": "keyword","ignore_above": 256}}}, "critics_consensus": {"type": "text"}, "directors": {"type": "text","fields": {"keyword": {"type": "keyword","ignore_above": 256}}}, "genre": {"type": "text","fields": {"keyword": {"type": "keyword","ignore_above": 256}}}, "in_theaters_date": {"type": "date","format": "iso8601"}, "movie_info": {"type": "text"}, "movie_title": {"type": "text","fields": {"keyword": {"type": "keyword","ignore_above": 256}}}, "on_streaming_date": {"type": "date","format": "iso8601"}, "poster_image_url": {"type": "text","fields": {"keyword": {"type": "keyword","ignore_above": 256}}}, "rating": {"type": "text","fields": {"keyword": {"type": "keyword","ignore_above": 256}}}, "rotten_tomatoes_link": {"type": "text","fields": {"keyword": {"type": "keyword","ignore_above": 256}}}, "runtime_in_minutes": {"type": "long"}, "studio_name": {"type": "text","fields": {"keyword": {"type": "keyword","ignore_above": 256}}}, "tomatometer_count": {"type": "long"}, "tomatometer_rating": {"type": "long"}, "tomatometer_status": {"type": "text","fields": {"keyword": {"type": "keyword","ignore_above": 256}}}, "writers": {"type": "text","fields": {"keyword": {"type": "keyword","ignore_above": 256}}} }} :::success ::: ---------------------------------- 使用功能查詢 ![](https://hackmd.io/_uploads/HJxqZ6o8_3.png) ![](https://hackmd.io/_uploads/HkeIaoLu3.png) :::success # Kibana ## Filter 篩選練習 :bulb: 1. 查看1980到2015年間的資料 (Jan 1, 1980 @ 00:00:00.000 ~ Dec 31, 2015 @ 23:59:59.000) 11358 2. 篩選nicolas cage所有的參演的電影 77 3. 觀眾評論(audience_count)不低於5000 8061 4. 篩選所有專業影評人評分(tomatometer_status)為Rotten的資料 6937 5. 篩選普通人的評分(audience_rating)不為空的資料 15667 ::: ## Logstash :::info /usr/share/logstash/bin/logstash -f /your/conf/path/xxx.conf EX1. LOGSTASH CONF SAMPLE input { stdin {} } output { stdout { codec => rubydebug } } Ex2. Logstash conf 輸出至 File input { stdin{} } output { file { path => "/home/bimap/output.log" } } EX3. LOGSTASH CONF 輸出至 ELASTICSEARCH input { stdin {} } output { elasticsearch { hosts => ["https://10.99.1.243:9200"] index => "logstash-test001-%{+YYYYMMdd}" ssl => true ssl_certificate_verification => false user => elastic password => "12345678" } stdout { codec => rubydebug } } ## Logstash Input plugin 接收 file 範例 input { file { path => "/path/to/file/*movie.csv" start_position => "beginning" sincedb_path => "/dev/null" } } TPC/UDP 接收 input { tcp { port => 514 } udp { port => 514 } } 接收 Beats 來源 log input { beats { port => 514 } } Logstash Output plugin 輸出到檔案 file { path => "/home/bimap/output.log" codec => line { format => "%{message}"} } Logstash Filter plugin CSV csv { separator => "," columns => ["rotten_tomatoes_link","movie_title","movie_info","critics_consensus","poster_image_url","rating"] } :::danger Date date { match =>["in_theaters_date","yyyy-MM-dd"] } Mutate - convert (改變資料型態) mutate { convert => { "tomatometer_rating" => "integer" } convert => { "audience_rating" => "integer" } } Mutate - split mutate { split => ["cast", ", "] } Mutate - add_field mutate { add_field => { "tomato" => "%{tomatometer_status}: %{tomatometer_rating}" } add_field => { "useless" => "^_^" } } Mutate - gsub mutate { gsub => [ "genre", " &", "," ] gsub => ["rating", "}", ""] } Mutate - replace mutate { replace => { "message" => "%{useless}: new message (￣个￣)" } } Mutate - remove mutate { remove_field => "[useless]" } Mutate - translate translate { source => "[tomatometer_status]" target => "[tomatometer_status_中文001]" dictionary => { "Rotten" => "差" "Certified Fresh" => "好" } fallback => "I'm a teapot" } # translate { field => "[tomatometer_status]" destination => "[tomatometer_status_中文002]" dictionary_path => '/home/bimap/tomato.yaml' } :::warning 補充 vim /home/bimap/tomato.yaml //內容 "Rotten": "差" "Fresh": "一般" "Certified Fresh": "好" ::: ## 主程式 :::info input { file { path => "/home/bimap/movie.csv" start_position => "beginning" sincedb_path => "/dev/null" } } filter { csv { separator => "," columns => ["rotten_tomatoes_link","movie_title","movie_info","critics_consensus","poster_image_url","rating","genre","directors","writers","cast","in_theaters_date","on_streaming_date","runtime_in_minutes","studio_name","tomatometer_status","tomatometer_rating","tomatometer_count","audience_status","audience_rating","audience_count","audience_top_critics_count","audience_fresh_critics_count","audience_rotten_critics_count"] } } output { elasticsearch { hosts => ["https://10.99.1.243:9200"] index => "logstash-movie-%{+YYYYMMdd}" ssl => true ssl_certificate_verification => false user => elastic password => "12345678" } stdout { codec => rubydebug } } :::