* Automatic parsing
* Good for simple log formats -> `Received <*> bytes from <*>`
* Does not work well with multiple successive parameters (Python lists, exceptions...)
* There are two different behaviours when it comes to parameter extraction.
1. Replacing non matching tokens with a placeholder -> Drain (Only usable when message and template have same length)
* Good for replacing simple parameters like `1` or `WAITING`
3. Use backtracking to replace multiple parameters with a single place holder -> Spell (Consumes to much information)
* Good for replacing parameters like `["tar", "cvf" "..."]`
* Data diversity is important to extract parameters from templates (Different data centers, ...)
* Parameter extraction requires more data then simply determining a template for a message
* Many parameters of rare log messages were not correctly extract in our test data set
* DeepLog requires a stable parser model. It can't handle newly mined templates because it did not learn on it. In theory it could simply ignore newly discovered templates, but if a new parameter gets extracted in a existing template DeepLog will no longer be able to make parameter predictions for it. To solve problem, a versioning scheme for log templates is required. This way the parser model can be improved in an online setting while the LSTM models stay the same.

* Anomaly detection
* Workflow
* Different dimensions of the hidden state and the amount of layers did not have a large effect on the evaluation metrics
* Even with complicated log datasets (~200 templates), we were able to train a model with less then 50MB data that generalize well (95% on test set)
* We believe by using a static threshold, instead of the top n approach, we reduced the detection of false positive by a margin
* Parameter
* Training and evaluating models for every template requires alot of time.
* The idea is to detect system slow down, for this parameters are not necessary. duration would be enough. Maybe train a single model which tries to predict only the duration.
* Unclear what to do with non numeric parameters
* We were not able to train models for most templates because we had to few samples. In some cases it also did not make sense to train models for extremly rare log events.

* Results on test data
* TODO Various parser errors on data from staging evironments.
* In-depth analysis would been easier on stable data (Application already in production for some time)
* You want to strike the right balance between size and diversity. Try to keep size as small as possible, diversity as high as possible.
* Workflow anomalies are easily understandable
* Parameter anomalies are hard to understand because the MSE normal distributions models the entire prediction of multiple time series. Maybe have a normal dist for every parameter of every model?

* Overall
* Toke make DeepLog operationally sustainable the training of models needs to be automated.
* The application changes over time and so the used log statements.
* Old log templates needs to be pruned from the parser model (can be trivially done by rembering when the template was last assigned to a message)
* The LSTM models might need retraining for bigger software releases. To retrain the model there need to be enough logs of the newest release. There might be time where DeepLog can be used directly after a release because not enough training data is available.
* Once the training script is in place it should be trivial to automatically train a model

We made a slight adjustement on how workflow anomalies are classified. In the
paper they check whether the incoming templates is in the top N predictions. We
decided to use a percentual threshold instead.

Let's say we have the following probability distribution `{ 0.94: t2, 0.01: t3,
0.01: t4, 0.04: t5}` and the template the most recent message is t4. When we
only classify the top 3 templates in the distribution, t4 would not be
classified as an anomaly.

Let's say we have the following probability distribution `{ 0.22: t2, 0.24: t3,
0.20: t4, 0.44: t5}` and the template the most recent message is t4. If we only
classify the top 3 templates in the distribution, t4 would wrongly classified as
an anomaly.