> The helm charts mentioned in this doc are primarily from [Prometheus Community](https://github.com/prometheus-community/helm-charts/)
## Prometheus Stack
The [kube-prometheus stack](https://github.com/prometheus-operator/kube-prometheus), a collection of Kubernetes manifests, [Grafana](http://grafana.com/) dashboards, and [Prometheus rules](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with [Prometheus](https://prometheus.io/) using the [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator).
### Installation
```bash
helm pull kube-prometheus-stack --repo https://prometheus-community.github.io/helm-charts
helm install prometheus-stack kube-prometheus-stack-48.3.1.tgz --create-namespace --namespace monitoring
```
* **Note:**
The release name is `prometheus-stack`. It will be referenced multiple times in this doc.
## Redis Exporter
### Installation
```
helm pull prometheus-redis-exporter --repo https://prometheus-community.github.io/helm-charts
helm install redis-exporter prometheus-redis-exporter-5.5.0.tgz --namespace redis -f redis-exporter.yaml
```
### Values file `redis-exporter.yaml`
```yaml
env:
- name: REDIS_EXPORTER_IS_CLUSTER
value: "true"
redisAddress: ml-redis-leader:6379
serviceMonitor:
enabled: true
namespace: monitoring
labels:
release: prometheus-stack
interval: 60s
timeout: 30s
auth:
enabled: true
secret:
name: ml-redis-auth
key: redis-password
```
### Grafana Dashboards
#### [redis-dashboard-for-prometheus-redis-exporter-1](https://grafana.com/grafana/dashboards/11835-redis-dashboard-for-prometheus-redis-exporter-helm-stable-redis-ha/)
*Currently used*
#### [redis-dashboard-for-prometheus-redis-exporter-2](https://grafana.com/grafana/dashboards/11692-redis-dashboard-for-prometheus-redis-exporter-1-x/)
## RabbitMQ Exporter
### Installation
```bash
helm pull prometheus-rabbitmq-exporter --repo https://prometheus-community.github.io/helm-charts
helm install rabbitmq-exporter prometheus-rabbitmq-exporter-1.8.0.tgz --namespace rabbitmq -f rabbitmq-exporter.yaml
```
The helm chart has a bug. The "rabbitmq.password" set in the values file doesn't take effect. Has to use a k8s Security. See the [link](https://github.com/prometheus-community/helm-charts/pull/3649) for details.
I modified the helm chart in order to make "rabbitmq.password" work. Why didn't use the Rabbitmq Security created by the rabbitmq Operator? Because somehow the username/password stored in the Security is the default ones but the ones I set in RabbitmqCluster, a k8s CR handled by the rabbitmq Operator. The Patch is as follows,
```diff
diff --git a/charts/prometheus-rabbitmq-exporter/templates/deployment.yaml b/charts/prometheus-rabbitmq-exporter/templates/deployment.yaml
index 7c4bfd0b..4b66890b 100644
--- a/charts/prometheus-rabbitmq-exporter/templates/deployment.yaml
+++ b/charts/prometheus-rabbitmq-exporter/templates/deployment.yaml
@@ -47,7 +47,7 @@ spec:
{{- if .Values.rabbitmq.configMapOverrideReference }}
- configMapRef:
name: {{ .Values.rabbitmq.configMapOverrideReference }}
- {{- end }}
+ {{- end }}
env:
{{- if .Values.rabbitmq.existingPasswordSecret }}
- name: RABBIT_PASSWORD
@@ -55,6 +55,9 @@ spec:
secretKeyRef:
name: "{{ Values.rabbitmq.existingPasswordSecret }}"
key: {{ Values.rabbitmq.existingPasswordSecretKey }}
+ {{- else if .Values.rabbitmq.password }}
+ - name: RABBIT_PASSWORD
+ value: {{ .Values.rabbitmq.password }}
{{- end }}
ports:
- containerPort: {{ .Values.service.internalPort }}
```
### Values file `rabbitmq-exporter.yaml`
```yaml
rabbitmq:
url: http://ml-rmq.rabbitmq:15672
user: ml-rabbitmq
password: s1t2c3b4@rabbitmq
capabilities: bert,no_sort
include_queues: ".*"
include_vhost: ".*"
skip_queues: "^$"
skip_verify: "false"
skip_vhost: "^$"
exporters: "exchange,node,overview,queue"
output_format: "TTY"
timeout: 30
max_queues: 0
excludeMetrics: ""
prometheus:
monitor:
enabled: true
interval: 60s
namespace:
- rabbitmq
additionalLabels:
release: prometheus-stack
```
### Grafana Dashboards
#### [The Dashboard for prometheus rabbitmq exporter](https://grafana.com/grafana/dashboards/4279-rabbitmq-monitoring/)
*Currently used*
#### [The Official Dashboard](https://grafana.com/grafana/dashboards/10991-rabbitmq-overview/)
*Currently used*
The official dashboard doesn't consume the data from rabbitmq exporter, but consumes the data directly from rabbitmq server.
It requires `rabbitmq-prometheus` to be enabled, a built-in plugin since [RabbitMQ v3.8.0](https://github.com/rabbitmq/rabbitmq-server/releases/tag/v3.8.0). The plugin is enabled by default in most of the cases.
```bash
# Enable the plugin
# Run the command within a rabbitmq container
rabbitmq-plugins enable rabbitmq_prometheus
```
So has to configure the `prometheus` to let it scrape data from rabbitmq server. Thus, a `ServiceMonitor` is going to be need (if Prometheus Operator is being used, which is just my case). The `ServiceMonitor` can be installed through the following command. See the [official link](https://www.rabbitmq.com/kubernetes/operator/operator-monitoring.html) for details.
```bash
curl -O https://raw.githubusercontent.com/rabbitmq/cluster-operator/main/observability/prometheus/monitors/rabbitmq-servicemonitor.yml
# Edit the manifest file, see the diff below
kubectl -n monitoring apply -f rabbitmq-servicemonitor.yml
```
```diff
--- rabbitmq-servicemonitor.yml.orig 2023-08-08 15:58:22.505454451 -0700
+++ rabbitmq-servicemonitor.yml 2023-08-08 16:02:14.368766617 -0700
@@ -3,7 +3,8 @@
kind: ServiceMonitor
metadata:
name: rabbitmq
- # If labels are defined in spec.serviceMonitorSelector.matchLabels of your deployed Prometheus object, make sure to include them here.
+ labels:
+ release: prometheus-stack
spec:
endpoints:
- port: prometheus
@@ -45,4 +46,5 @@
matchLabels:
app.kubernetes.io/component: rabbitmq
namespaceSelector:
- any: true
+ matchNames:
+ - rabbitmq
```
* **Note:**
The label `release: prometheus-stack` is very important. Until the label is added, the `ServiceMonitor` won't work.
Why?
Run the following command to check the spec the CR `Prometheus`
```bash
kubectl -n monitoring get prometheus prometheus-stack-kube-prom-prometheus -o yaml
```
And note the part as follows,
```yaml
spec:
serviceMonitorNamespaceSelector: {}
serviceMonitorSelector:
matchLabels:
release: prometheus-stack
```
It means the Prometheus Operator will keep watching the `ServiceMonitor` in any namespaces and only pay attention to those having label `release: prometheus-stack`.
## ElasticSearch Exporter
### Installation
```bash
helm pull prometheus-elasticsearch-exporter --repo https://prometheus-community.github.io/helm-charts
helm install elasticsearch-exporter prometheus-elasticsearch-exporter-5.2.0.tgz --namespace opensearch -f elasticsearch-exporter.yaml
```
### Values file `elasticsearch-exporter.yaml`
```yaml
env:
ES_USERNAME: ml-elastic
extraEnvSecrets:
ES_PASSWORD:
secret: opensearch-extra-admin-password
key: password
es:
uri: https://ml-os:9200
sslSkipVerify: true
all: true
indices: true
indices_settings: true
indices_mappings: true
aliases: false
shards: true
snapshots: true
cluster_settings: false
slm: false
data_stream: false
timeout: 30s
serviceMonitor:
enabled: true
namespace: monitoring
labels:
release: prometheus-stack
interval: 30s
scrapeTimeout: 10s
```
### Grafana Dashboards
#### [ElasticSearch](https://grafana.com/grafana/dashboards/6483-elasticsearch/)
*Currently used*
Bugs
- For "cluster health", it displays "N/A"
#### [Elasticsearch Exporter Quickstart and Dashboard](https://grafana.com/grafana/dashboards/14191-elasticsearch-overview/)
*Currently used*
Similar to the upper one and with more recent update
#### [Elasticsearch - Index Stats](https://grafana.com/grafana/dashboards/13072-elasticsearch-index-stats/)
*Currently used*
#### [Elasticsearch Cluster - Indices](https://grafana.com/grafana/dashboards/3598-elasticsearch-cluster-indices/)
> This dashboard monitors a cluster using the data collected through the x-pack monitoring collector.
??? I haven't figured out how to add x-pack data source for this dashboard.
## OpenCTI Server
Two kinds of Metrics
- General NodeJS metrics
There are plenty of third-party Grafana Dashboards (see the [Grafana Dashboards](#Grafana-Dashboards3) below).
- OpenCTI specific metrics
There is no any third-party Grafana Dashboards.
Have to create on my own.
Access `<http://<opencti-server-exporter>:14269/metrics` to find out what OpenCTI specific metrics there are.
### How a NodeJS Application exposes its metrics
The NodeJS metrics works this way
```mermaid
graph LR;
a("NodeJS Application<br/>(Also a Prometheus Exporter)") --> |scraped by| b(Prometheus) --> |fetched by| c(Grafana) --> |displayed in| d(A Dashboard)
```
I.e., it works as long as the Nodejs Applications make themselves the Prometheus Exporter. AFAIK, there are two ways for a NodeJS Application to be a Prometheus Exporter, one is to use [Prom-Client](https://www.npmjs.com/package/prom-client), another is to use [Express Prometheus Middleware](https://www.npmjs.com/package/express-prometheus-middleware).
### Enable Prometheus Exporter of OpenCTI Server
Via configuration file
```json
"app": {
"telemetry": {
"metrics": {
"enabled": true,
"exporter_prometheus": 14269
}
}
}
```
Via environment variables
```bash
APP_TELEMETRY__METRICS__ENABLED="true"
APP_TELEMETRY__METRICS__EXPORTER_PROMETHEUS="14269"
```
### Kubernetes Resources
#### The changes of `Service`
*Used to open the exporter port for Prometheus*
Example
```yaml
apiVersion: v1
kind: Service
metadata:
...
spec:
ports:
- name: prometheus
port: 14269
targetPort: 14269
```
The `prometheus` port can be added in the existing OpenCTI `Service`.
Or create a new `Service` for the exporter only, if the existing OpenCTI `Service` is a public-facing service (e.g., of type `NodePort` and `LoadBalancer`), and you don't want to expose the `prometheus` port to the public.
#### A new `ServiceMonitor`
*Used to configure Prometheus to scrape data from OpenCTI*
Example
```yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
app: opencti-server
app.kubernetes.io/component: opencti-server
prometheus-exporter: opencti-server
release: prometheus-stack
name: opencti-prometheus-exporter-opencti-server
namespace: monitoring
spec:
endpoints:
- honorLabels: true
port: prometheus
jobLabel: opencti
namespaceSelector:
matchNames:
- opencti
selector:
matchLabels:
app: opencti-server
app.kubernetes.io/component: opencti-server
prometheus-exporter: opencti-server
```
### Grafana Dashboards
#### [Node.js and Express Metrics](https://grafana.com/grafana/dashboards/14565-node-js-dashboard/)
*Currently Used*
Work with the Nodejs Applications that uses [Express Prometheus Middleware](https://www.npmjs.com/package/express-prometheus-middleware) to turn themselves into the Prometheus Exporter.
Some visual panel doesn't work (has no data to display), because the current OpenCTI is not using [Express Prometheus Middleware](https://www.npmjs.com/package/express-prometheus-middleware) any more. (It was using it, here is the [evidence](https://github.com/OpenCTI-Platform/opencti/pull/1598/files), take look at the diff of file `package.json`)
#### [NodeJS Application Dashboard](https://grafana.com/grafana/dashboards/11159-nodejs-application-dashboard/)
*Currently Used*
*The most downloaded nodejs dashboards*
Work with the Nodejs Applications that uses [Prom-Client](https://www.npmjs.com/package/prom-client) to turn themselves into the Prometheus Exporter.
## OpenCTI Worker
### Enable Prometheus Exporter of OpenCTI Worker
Via environment variable
```bash
WORKER_TELEMETRY_ENABLED="TRUE"
WORKER_PROMETHEUS_TELEMETRY_PORT="14270"
```
### Kubernetes Resources
#### A new `Service`
```yaml
apiVersion: v1
kind: Service
metadata:
labels:
app: opencti-worker
app.kubernetes.io/component: opencti-worker
prometheus-exporter: opencti-worker
name: opencti-prometheus-exporter-opencti-worker
namespace: opencti
spec:
ports:
- name: prometheus
port: 14270
protocol: TCP
targetPort: 14270
selector:
app: opencti-worker
app.kubernetes.io/component: opencti-worker
app.kubernetes.io/instance: opencti
app.kubernetes.io/name: opencti
```
#### A new `ServiceMonitor`
```yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
app: opencti-worker
app.kubernetes.io/component: opencti-worker
prometheus-exporter: opencti-worker
release: prometheus-stack
name: opencti-prometheus-exporter-opencti-worker
namespace: monitoring
spec:
endpoints:
- honorLabels: true
port: prometheus
jobLabel: opencti
namespaceSelector:
matchNames:
- opencti
selector:
matchLabels:
app: opencti-worker
app.kubernetes.io/component: opencti-worker
prometheus-exporter: opencti-worker
```
### Grafana Dashboards
No any predefined (third-party of official) Grafana Dashboards available.
Have to create on my own.
Access `http://<opencti-worker-exporter-service>:14270/metrics` to find out what metrics are exposed.
Sign in with Wallet
Connect another wallet