<style>
b { color: #94e5ff }
g { color: #777777 }
</style>

# Watcher alerts

[< Back](../../home.md)

## Query DSL

:information_source: official documentation: [Elasticsearch Guide](https://www.elastic.co/guide/en/elasticsearch/reference/7.16/index.html)

**D**omain **S**pecific **L**anguage based on JSON to define queries, consisting of two types of clauses:

- **Leaf query clauses** - look for particular value in a particular field (<b>match</b>, <b>term</b>, <b>range</b>, ...)
- **Compound query clauses** - wrap other leaf or compound queries to combine multiple queries ([bool](#bool-query), <b>dis_max</b>)

### Most useful Leaf query clauses
:information_source: You can experiment or validate your own queries on [Elastic Console](https://154b1b4905dc4c8eb06aa13fb7f2329e.eu-central-1.aws.cloud.es.io:9243/app/dev_tools#/console)
- **term** - Returns documents that contain an exact term in a provided field. Suitable for precise values such as a price, IDs, usernames...
:heavy_exclamation_mark: Avoid using a term query for <b>text</b> fields.

    GET _search
    {
      "query": {
        "term": {"integrationId": 16091}
      }
    }

- **match** - Returns documents that match a provided text, number, date or boolean value.

    GET integrations-services-prod-2022.02.17/_search
    {
      "query": {
        "match": {"message": "webhook"}
      }
    }

:information_source: returns all records where the message will contain the word *webhook*, (e.g. "undefining webhook", "webhook signature mismatch!", ...)
- **range** - Returns documents that contain terms within a provided range.
\- Optional parameters: **gt**, **gte**, **lt**, **lte**,

    GET integrations-services-prod-2022.02.17/_search
    {
      "query": {
        "range": {
          "@timestamp": {
            "gte": "now-2h",
            "lt": "now-1h"
          }
        }
      }
    }

- **query string** - Returns documents based on a provided query string, using a parser with a strict syntax. This query uses a syntax to parse and split the provided query string based on operators, such as AND or NOT.

**Field names**:
- **<g>label:error</g>** - lable field contains error
- **<g>message:error AND NOT httpBase</g>** - message field contains error but not httpBase
- **<g>response.\\*:(InvalidValue OR RecordInvalid)</g>** - any of the fields response.data, response.result, response.code, ... contains InvalidValue or RecordInvalid

**Wildcards**:
- **<g>?</g>** - to replace single character
- **<g>*</g>** - to replace zero or more characters

    GET integrations-services-prod-2022.02.17/_search
    {
      "query": {
        "query_string": {
          "query": "response.\\*:(*Value AND Record*)"
        }
      }
    }

## Compound queries
### bool query
A query that matches documents matching boolean combinations of other queries. It is built using one or more boolean clauses, each clause with a typed occurrence. The occurrence types are:
- **must** - query must appear in matching document and contribute to the score (not relevant for us). It behave like logical AND operation.
e.g. Find all errors, where message contains "createOrUpdateContact" and integration type is "hubspot"
*KQL*: `message: "createOrUpdateContact" and integrationType: "hubspot"`

    GET integrations-services-prod-2022.02.17/_search
    {
      "query": {
        "bool": {
          "must": [
            { "match": {
              "message": "createOrUpdateContact"
            }},
            { "match": {
              "integrationType": "hubspot"
            }}
          ]
        }
      }
    }

- **filter** - Similar to <u>must</u>. Query must appear in matching document, but will not contribute to the score.
e.g. Previous example using a filter.

  GET integrations-services-prod-2022.02.17/_search
  {
    "query": {
      "bool": {
        "filter": [
          { 
            "match": { "message": "createOrUpdateContact" }
          },
          {
            "match": { "integrationType": "hubspot" }
          }
        ]
      }
    }
  }

- **should** - Query should appear in the matching document. It behaveves like logical OR operation.

  GET integrations-services-prod-2022.02.17/_search
  {
    "query": {
     "bool": {
        "should": [
          { 
           "match": { "integrationType": "gont" }
          },
          {
            "match": { "integrationType": "tray" }
          }
        ],
        "minimum_should_match": 1 
      }
    }
  }

- **must_not** - Query must not appear in the matching documents.
*KQL*: `message: "createOrUpdateContact" and integrationType: "hubspot" and not @timestamp <= "now-10m"`

  GET integrations-services-prod-2022.02.17/_search
  {
    "query": {
      "bool": {
        "must": [
          { "match": {
            "message": "createOrUpdateContact"
          }},
          {"match": {
            "integrationType": "hubspot"
          }}
        ],
        "must_not": [
          {"range": {
            "@timestamp": {
              "lte": "now-10m"
            }
          }}
        ]
      }
    }
  }

# Watcher

You can use Watcher to watch for changes or anomalies in your data and perform the necessary actions in response.

:information_source: You find Watcher at this [link](https://154b1b4905dc4c8eb06aa13fb7f2329e.eu-central-1.aws.cloud.es.io:9243/app/management/insightsAndAlerting/watcher/watches), or alternatively go to:
**elastic (Kibana)** -> Side Navigation -> **Management** -> Stack Management ->
**Alerts and Insights** -> Watcher

:information_source: Type <u>integrations</u> to search bar to display only the list of integration watchers

## What are the goals
- have an overview of the state of integrations
- know about the problem in advance (before the customer reports it)

## What to avoid
- we don't want to receive :100: notifications in :one: hour about the call export failed

## How it works

A watch is constructed from four simple building blocks:
- **Schedule** - A schedule for running a query and checking the condition.
- **Query** - The query to run as input to the condition.
- **Condition** - A condition that determines whether or not to execute the actions. Simple condition or scripting are available.
- **Actions** - One or more actions, such as sending email, pushing data to 3rd party systems. In our case, it's sending notifications to slack channel.

## Createing new advanced watch
Define name in the format "TBD"
The basic structure will be automatically generated.

{
  "trigger": {
    "schedule": {
      "interval": "30m"
    }
  },
  "input": {
    "search": {
      "request": {
        "body": {
          "size": 0,
          "query": {
            "match_all": {}
          }
        },
        "indices": [
          "*"
        ]
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gte": 10
      }
    }
  },
  "actions": {
    "my-logging-action": {
      "logging": {
        "text": "There are {{ctx.payload.hits.total}} documents in your index. Threshold is 10."
      }
    }
  }
}

### Trigger
Controls how often a watch is triggered. There are several alternatives. We can use hour schedule, daily, weekly, monthly, cron schedule or interval schedule. More info [here](https://www.elastic.co/guide/en/elasticsearch/reference/7.16/trigger-schedule.html). In example above is watcher periodically triggered every 30 minutes.

### Input
Load data into the watch payload, based on a defined query.
There are four input types. We will be mainly interested in:
- **<g>search</g>**: load the results of a search into the execution context (showen in basic structure above).
- **<g>chain</g>**: use a series of inputs to load data into the execution context.

  "input": {
    "chain": {
      "inputs": [
        {
          "first": {
            "search": {
              "request": {
                "indices": [
                  "integrations*prod*"
                ],
                "rest_total_hits_as_int": true,
                "body": {
                  "query": {
                    "bool": {
                      "must": [
                        {
                          "query_string": {
                            "query": """message: "Export failed" AND model: "Cdr" """
                          }
                        },
                        {
                          "range": {
                            "@timestamp": {
                              "gte": "now-1h"
                            }
                          }
                        }
                      ]
                    }
                  },
                  "_source": [
                    "message"
                  ]
                }
              }
            }
          }
        },
        {
          "second": {
            "search": {
              "request": {
                "indices": [
                  "integrations*prod*"
                ],
                "rest_total_hits_as_int": true,
                "body": {
                  "query": {
                    "bool": {
                      "must": [
                        {
                          "query_string": {
                            "query": """message: "Export failed" AND model: "Cdr" """
                          }
                        },
                        {
                          "range": {
                            "@timestamp": {
                              "gte": "now-2h",
                              "lt": "now-1h"
                            }
                          }
                        }
                      ]
                    }
                  },
                  "_source": [
                    "message"
                  ]
                }
              }
            }
          }
        }
      ]
    }
  }

- **first**, **last** are the names we will refer to in the condition
- **rest_total_hits_as_int** if true, response return hits.total as integer. If false, it return hits.total as object. Default is false. Check response difference in [Elastic Console](https://154b1b4905dc4c8eb06aa13fb7f2329e.eu-central-1.aws.cloud.es.io:9243/app/dev_tools#/console)
- **indices** search for query in the *integrations\*prod\** index and loads the response into the watch payload.
- **body** contains the query definition we discussed at the beginning [Query DSL]( #query-dsl)
- **_source** need to check difference between with and without

### Conditions
- **compare**: perform simple comparisons against values in the watch payload

    "condition": {
      "compare": {
        "ctx.payload.first.hits.total": {
          "gte": "{{ctx.payload.second.hits.total}}"
        }
     }
    }

- **script**: use a script to determine whether or not to execute the watch actions.

    "condition": {
      "script": {
        "source": "return ((ctx.payload.first.hits.total + 0.01) / ctx.payload.second.hits.  total) > 1.10",
        "lang": "painless"
      }
    }

Default script language is [*painless*](https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-painless.html). Painless has a simple syntax. Support arithmetic, relational, logical operations like in common languages.

### Action
We are using webhook action to send notifications to slack chanels.
**prod-allie-test** chanel is used for testing purposes.
- `webhook.path: "services/TBMUGUCDP/B02RVK737SQ/UbqvtNLhf1Ti6lL7IMK9m19i"`

**dev-reporting-integrations** chanel for production integrations reports
- `webhook.path: "services/TBMUGUCDP/B02RVK737SQ/LQj5FHw1ZLTfhxBg5HJmYSBh"`

"actions": {
  "my_webhook": {
    "webhook": {
      "scheme": "https",
      "host": "hooks.slack.com",
      "port": 443,
      "method": "post",
      "path": "services/TBMUGUCDP/B02RVK737SQ/UbqvtNLhf1Ti6lL7IMK9m19i",
      "params": {},
      "headers": {
        "Content-Type": "application/json"
      },
      "body": """{
        "blocks": [
          {
            "type": "section",
            "text": {
              "type": "mrkdwn",
              "text": "*Integrations LUKAS TEST ALERT compare last two hours* - {{ctx.payload.first.hits.total}} > {{ctx.payload.second.hits.total}} "
            },
            "accessory": {
              "type": "button",
              "text": {
                "type": "plain_text",
                "text": "Error Logs",
                "emoji": true
              },
              "value": "error_logs",
              "url": "https://154b1b4905dc4c8eb06aa13fb7f2329e.eu-central-1.aws.cloud.es.io:9243/app/kibana#/discover/dec1d130-7919-11ec-8a40-d1559359c704?_g=(refreshInterval%3A(pause%3A!t%2Cvalue%3A0)%2Ctime%3A(from%3Anow-1h%2Cto%3Anow))",
              "action_id": "button-action"
            }
          }
        ]
      }"""
    }
  }
}

### Simulation
After setting up the watcher, we can simulate it.
- Click on the pencil icon :pencil2: and then select <u>Simulate tab</u>.
- Select checkbox <u>*Ignore condition*</u>
- Change Action mode for a slack webhook to <u>*force_execute*</u>
- Fire the <u>*Simulate watch*</u> button

##### TODO
- ? aggregations and array_compare