---
# System prepended metadata

title: Metrics and monitoring

---

# Metrics and monitoring


## Why to monitor?

- to prevent outages
- analyse performance (see below what to monitor)
- predict future usage (e.g. how many more instances we can run using current resources)


## What to monitor

### Docker hosts

- CPU utilization per container
- memory utilization per container
- network throughput (in/out) per container

### Tomcat

- log file size for unexpected entries
- JVM memory allocation (heap available/allocated)
- number of threads

### Web applications (servlets)

- request rate per endpoint
- request time per endpoint
- request/response size per endpoint
- cache hits/misses per cache
- cache allocated size and disk space for persisted caches

### Servers

- filesystem available disk space
- CPU utilization
- memory utilization


## Alerts

- log files grow rate is more than threshold (n Mb per hour)
- host CPU utilization is above threshold (e.g. > 80%)
- host memory utilization is above threshold (e.g. > 90%)
- swap/temp fs?
- web API availability (e.g. no metrics for period of more than 5 minutes)
- job runner stopped working

## Open questions

- can we combine URLs that are monitored in the metrics, e.g. all `/apps/<appName>/config` URLs?
- can we use monitoring (Grafana) for usage statistics?
- how long to persist monitoring data? (e.g. last 3 months)
- how much traffic to allow it generate? (to calculate intervals)
- how to secure exposing metrics from different sources?
- can we setup prometheus/statistics per server instance, not per servlet?


## Current setup

### Central components

Setup taken originally from https://github.com/stefanprodan/dockprom
Contains main componets:
- prometheus (fetch and store timeseries data)
- grafana (visualization of timeseries data sources)
- alertmanager
- pushgateway
- caddy as reverse proxy

### Host components

And 2 exporters:
- nodeexporter (exports metrics from host OS)
- cadvisor (exports metrics from docker containers)

### Application components

- Dependency https://github.com/prometheus/client_java
- Metrics filter [parameters](https://github.com/prometheus/client_java/blob/master/simpleclient_servlet/src/main/java/io/prometheus/client/filter/MetricsFilter.java#L30)


# Actions (TODO)

- reduce size of metrics payload by reducing number of segments to collect statistics from web apps (and make it configurable if possible)
- answer how much space Prometheus use currently? what it the retention period for its DB cleanup? where it is persisted?
- put configuration for current setup to Github repository
- look into other DBs that support not only polling, but pushing (to avoid metrics showing up in logs)
- try to filter out metrics from Tomcat access logs (server configuration if possible e.g. exclude URL pattern `^.+\/metrics$` from logs)
- implement push-strategy in jobs (decide upon which metrics to collect)

