# Prometheus integration

We have [prometheus integration](https://github.com/elastic/integrations/tree/main/packages/prometheus), that supports collecting metrics:
- from Prometheus endpoint (`collector` data_stream) to get metrics, exposed by the service in prometheus format
- from Prometheus server using `remote_write` (`remote_write` data_stream) to get metrics, that prometheus server collects from various endpoints it scrapes
- using PromQL (`query` data_stream)

## How to setup and test different prometheus configuration

1. Install prometheus server using helm:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus-server prometheus-community/prometheus

### Scraping metrics from Prometheus endpoint on kubernetes

`prometheus.collector` works for both managed and standalone elastic-agent.
**Managed elastic-agent**
1. Enabled `Prometheus collector metrics` configuration
2. Use prometheus service as a host value, example: `prometheus-server-server.default:80`.
3. Adjust `FLEET_URL` and `FLEET_ENROLLMENT_TOKEN` and apply [manifest](https://github.com/elastic/elastic-agent/blob/main/deploy/kubernetes/elastic-agent-managed-kubernetes.yaml).

**Standalone configuration**
1. Use [standalone manifest](https://github.com/elastic/elastic-agent/blob/main/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml) and replace `inputs`, defined in `agent.yml`:

inputs:
  - name: prometheus
    type: prometheus/metrics
    use_output: default
    meta:
      package:
        name: prometheus
        version: 0.7.0
    data_stream:
      namespace: default
    streams:
      - data_stream:
          dataset: prometheus.collector
          type: metrics
        period: 10s
        hosts:
          - 'prometheus-server-server.default:80'
        metricsets:
          - collector
        # ssl.certificate_authorities:
        #   - /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
        use_types: true
        # bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        # metrics_filters.exclude: null
        # metrics_filters.include:
        #   - prometheus_tsdb_*
        #   - prometheus_http_*
        rate_counters: true
        # metrics_path: /metrics
        # password: changeme
        # username: elastic

In case there are multiple pods of the prometheus server and all pods should be scraped individually (example: prometheus server deployment has 2 replicas): now it is only possible with elastic-agent standalone. To use predefined integration (prometheus integration), should be used autodiscover like in this example - [doc](https://www.elastic.co/guide/en/fleet/current/running-on-kubernetes-standalone.html#_autodiscover_targeted_pods).

To run elastic-agent standalone to scrape each pod of prometheus service individually [elastic-agent standalone](https://github.com/elastic/elastic-agent/blob/main/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml) should be deployed with next input configuration:

inputs:
  - name: prometheus
    type: prometheus/metrics
    use_output: default
    meta:
      package:
        name: prometheus
        version: 0.7.0
    data_stream:
      namespace: default
    streams:
      - data_stream:
          dataset: prometheus.collector
          type: metrics
        metricsets:
          - collector
        hosts:
          - '${kubernetes.pod.ip}:9090'
        period: 10s
        use_types: true
        rate_counters: true
        # metrics_path: /metrics
        # metrics_filters.include:
        #   - prometheus_tsdb_*
        #   - prometheus_http_*
        # metrics_filters.exclude:
        #   - prometheus_api_*
        #   - go_*
        # username: "user"
        # password: "secret"
        #bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        #ssl.certificate_authorities:
        #  - /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
        condition: ${kubernetes.container.name} == 'prometheus-server'

All metrics exposed by the prometheus server `/metrics` endpoint will be scraped by default.
Additionally you can modify extracted scraped metrics by configuring `metrics_filters`

### Scraping metrics from Prometheus server using remote_write on kubernetes

When prometheus.remote_write metricset is enabled - metricbeat starts listening on the defined in the configuration `host`:`port` (default: `localhost:9201`). Prometheus must use this address as a `remote_write` URL.

`prometheus.remote_write` works with both managed and standalone elastic-agent, the only missing part for both - we do not have a `Service` for elastic-agent to be able to refer it as a remote_write URL (in prometheus config).

Prometheus Remote_Write:

**Managed elastic-agent**
1. Add Prometheus Integration configuration `Prometheus remote write metrics`: set host `0.0.0.0`
2. Use next [manifest](https://github.com/elastic/elastic-agent/blob/main/deploy/kubernetes/elastic-agent-managed-kubernetes.yaml) and add kubernetes `Service` resource:

---
apiVersion: v1
kind: Service
metadata:
  name: elastic-agent
  namespace: kube-system
  labels:
    app: elastic-agent
spec:
  ports:
    - port: 9201
      protocol: TCP
      targetPort: 9201
  selector:
    app: elastic-agent
  sessionAffinity: None
  type: ClusterIP

2. Adjust `FLEET_URL` and `FLEET_ENROLLMENT_TOKEN` and apply manifests.
4. Update prometheus release with modified configuration

helm upgrade --install prometheus-server prometheus-community/prometheus -f values.yml

$ cat values.yml
server:
  remoteWrite:
    - url: http://elastic-agent.kube-system:9201/write

**Standalone configuration**
1. Use [standalone manifest](https://github.com/elastic/elastic-agent/blob/main/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml) and replace `inputs`, defined in `agent.yml`:

inputs:
- name: prometheus
  type: prometheus/metrics
  use_output: default
  meta:
    package:
      name: prometheus
      version: 0.7.0
  data_stream:
    namespace: default
  streams:
    - data_stream:
        dataset: prometheus.remote_write
        type: metrics
      metricsets:
        - remote_write
      host: '0.0.0.0'
      port: 9201

### Scraping metrics using PromQL on kubernetes

**Managed elastic-agent**
1. Enable `Prometheus query metrics`
2. Set prometheus service as a Host value, example: `prometheus-server-server.default:80`.
3. Adjust `FLEET_URL` and `FLEET_ENROLLMENT_TOKEN` and apply [manifest](https://github.com/elastic/elastic-agent/blob/main/deploy/kubernetes/elastic-agent-managed-kubernetes.yaml).

**Standalone configuration**
1. Use [standalone manifest](https://github.com/elastic/elastic-agent/blob/main/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml) and replace `inputs`, defined in `agent.yml`:

inputs:
- name: prometheus
  type: prometheus/metrics
  use_output: default
  meta:
    package:
      name: prometheus
      version: 0.7.0
  data_stream:
    namespace: default
  streams:
    - data_stream:
          dataset: prometheus.query
          type: metrics
      metricsets:
        - query
      hosts:
          - 'prometheus-server-server.default:80'
        period: 10s
        queries:
          - name: instant_vector
            params:
              query: 'sum(rate(prometheus_http_requests_total[1m]))'
            path: /api/v1/query

Metrics will be stored under `prometheus.query.*`.