# ELK Stack

ELK (Elasticsearch/Logstash/Kibana) 是一套完整的數據分析工具

## [Filebeat](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-overview.html)

- type: log

  # Change to true to enable this input configuration.
  enabled: true
  
  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - L:\Logs\RTX\*\*\All*.log*
    
  # Optional additional fields. These fields can be freely picked
  # to add additional information to the crawled log files for filtering
  fields:
    source_type: "log4net"
    index_field: "general"
  
  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  exclude_files: ['zip$','7z$','\b(\w*[Jj]obService\w*)\b']
  
  #Specifies the regular expression pattern to match.
  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}'
  #Defines whether the pattern is negated. The default is false.
  multiline.negate: true
  #Specifies how Filebeat combines matching lines into an event.
  multiline.match: after
  
  #If this option is enabled, Filebeat ignores any files that were modified before the specified timespan.
  ignore_older: 24h

## [LogStash](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-overview.html)

if [index_prefix] =~ "general" {
		grok{
			break_on_match => false
			match => [ "message", "%{TIMESTAMP_ISO8601:log_timestamp} \[%{NUMBER:thread:int}\] %{DATA:log_type} %{DATA:logger} %{GREEDYDATA:log_body}" ]
			match => [ "message", "(?i).*DBManager2.*- (?<database>\[.*\])(?<schema>\[.*\])\.(?<storedProcedure>\[.*\]) " ]
			match => [ "log_body", "^\[%{DATA:session_id}\] - (?<summary>[^\n]*)\n{0,1}(?<detail>.*)" ]
			match => [ "log_body", "^- (?<summary>[^\n]*)\n{0,1}(?<detail>.*)" ]

			match => [ "summary", "(?i)\[PacketMonitor\]\[%{DATA:packetType}\]" ]
			match => [ "summary", "(?i)(url|requestUrl):\s*%{DATA:url}(\?|\]|,)" ]
			match => [ "summary", "(?i)query:\s*%{DATA:query}(\]|,)" ]
			match => [ "summary", "(?i)topic(:|=)\s*%{DATA:messagingTopic}(\]|,)" ]

			match => [ "summary", "(?i)requestSize:\s*%{DATA:requestSize}\s*bytes" ]
			match => [ "summary", "(?i)responseSize:\s*%{DATA:responseSize}\s*bytes" ]
			match => [ "summary", "(?i)requestSize:\s*%{NUMBER:requestBytes:int}(,|\])" ]
			match => [ "summary", "(?i)responseSize:\s*%{NUMBER:responseBytes:int}(,|\])" ]
			match => [ "summary", "(?i)channel:\s*%{DATA:channel}(,|\])" ]

			match => [ "summary", "(?i)(spantime|duration):\s*%{TIME:duration}(\]|,)*" ]
			match => [ "summary", "(?i)spendtime:\s*%{NUMBER:spendtime:float}(,|\])" ]
			match => [ "summary", "\[%{TIME:duration}\]" ]
			match => [ "summary", "(?i).*\[(?<slowType>.*slow)\]" ]

			match => [ "message", "(?i)(ip|Forwarded-For).*(:|=)\s*\"*%{IP:ip}\"*(,|\]| )*" ]
		}
	}

## [Index Template](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-templates.html)

rtx-index-general-template
{
  "index": {
    "lifecycle": {
      "name": "dng-lifecycle-policy",
      "rollover_alias": "rtx-index-general"
    },
    "refresh_interval": "10s",
    "number_of_shards": "1",
    "number_of_replicas": "1"
  }
}

PUT rtx-index-general-000001
{
    "aliases": {
    "rtx-index-general":{
        "is_write_index":true
    }
  }  
}

## [Index Pattern](https://www.elastic.co/guide/en/kibana/7.17/index-patterns.html)
Index pattern can point to one or more indices.
rtx-index-general-* => rtx-index-general-000001、rtx-index-general-000002 ...

## [Kibana Watcher](https://www.elastic.co/guide/en/kibana/current/watcher-ui.html)

{ //多久觸發Watcher
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "rtx-index-fatal-*"
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "query": {
            "bool": {
              "filter": [
                {　／／搜尋的時間範圍
                  "range": {
                    "@timestamp": {
                      "gte": "now-1m"
                    }
                  }
                }
              ],
              "boost": 1
            }
          },
          "aggs": {
            "partners": {
              "terms": {
                "field": "partner.keyword"
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "script": {
      "source": "def result = ctx.payload.aggregations.partners.buckets; if (result.size() == 0) return false; return result.get(0).doc_count > 0;",
      "lang": "painless"
    }
  },
  "actions": {
    "skype_notify": {
      "transform": {
        "script": {
          "source": "return ctx.payload.aggregations.partners.buckets.stream().filter(op -> op.doc_count > 0).collect(Collectors.toList());",
          "lang": "painless"
        }
      },
      "webhook": {
        "scheme": "http",
        "host": "172.24.3.240",
        "port": 888,
        "method": "get",
        "path": "/",
        "params": {
          "msg": """Job Retry Failed
=================================
Partners who has Fatal Event Failed within 1 minutes are as following. 
		
{{#ctx.payload._value}}{{key}} : {{doc_count}}
{{/ctx.payload._value}}  
		
		
Link : http://kibana.mrk6.com/s/rtx/goto/2f901869530e21bf881c23a18fe29816
=================================""",
          "to": "ALERT_RTX"
        },
        "headers": {}
      }
    }
  }
}

[John Wu有寫手把手建置ELK的Blog 推推](https://blog.johnwu.cc/article/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-centos-red-hat.html)

# ELK advance notification

**Python API的組成由Flask 框架 + Python Skype套件( skpy )構成**

[Flask](https://zh.wikipedia.org/wiki/Flask)

from flask import Flask, jsonify, request

@app.route('/sendPicture', methods=['GET'])
@log_request
def sendPicture():
    data = request.get_json()     
    url = data['url']
    fileName = datetime.now().strftime("%Y-%m-%d_%H%M%S")
    skype_agent = skypeHelper.skype_agent(const.config_info,const.config_info['ChatName'])
    Selenium_Helper.execute(url, fileName) 
    filePath = f"{const.config_info['outputPath']}\\{fileName}.png"
    with open(rf"{filePath}", "rb") as f:
        skype_agent.sendFile(f, filePath, image=True)  
        return jsonify(isError=False, message="Success", statusCode=200)

if __name__ == "__main__":
    app.run(host=const.config_info['Host'], port=const.config_info['Port'], debug=True)

[skpy](https://pypi.org/project/SkPy/)

from skpy import Skype, SkypeChats

sk = Skype(config_info["account"], config_info["password"])

# 建立 SK 物件
skc = SkypeChats(sk)

# 指定發送訊息的目的聊天室(可以是群組 or 個人)
# cht_name = "Bot Test"

# 搜尋聊天室 ID
target_id = search_sk_id(sk, cht_name)

# 聊天功能匯入「聊天室ID」
ch = sk.chats[target_id]

return ch