# PoC: Replace Prometheus Exporters by OpenTelemetry Receivers in Gardener Monitoring Stack

**Hackers**: @Plamen Kokanov, @Stoyan Vitanov

**Description**: The Gardener monitoring stack uses various upstream Prometheus receivers to collect typical metrics of a Kubernetes cluster. This includes, for instance, `node-exporter`, `kube-state-metrics`, or `cAdvisor`. The OpenTelemetry project introduced the receiver concept in the Collector component to gather telemetry data. The opentelemetry-collector-contrib GitHub repository already contains multiple receivers that serve a similar purpose as the mentioned Prometheus exporters.

This hackathon project should build a PoC setup, purely based on native OpenTelemetry components, to monitor a Kubernetes cluster and gather experience about the best telemetry data sources when implementing the Observability 2.0 strategy. Potential gaps in the metrics provided by the OpenTelemetry receivers should be identified, and the two approaches to fetch telemetry data should be compared.

**Branch with work done during the hackathon**: https://github.com/vitanovs/gardener/tree/feat/hackathon

## Metrics pipeline setup

During the hackathon we managed to replace `node-exproter` with the [`hostmetrics`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/hostmetricsreceiver/README.md) receiver, the `cAdvisor` prometheus scrape config with the [`kubeletstats`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/kubeletstatsreceiver/README.md) receiver, and `kube-state-metrics` with the [`k8scluster`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/k8sclusterreceiver/README.md) receiver.

The `hostmetrics` and `kubeletstats` recievers were added to the otel-collector service that runs on shoot nodes. This service was added as part of [`GEP-34`](https://github.com/gardener/gardener/blob/master/docs/proposals/34-observability2.0-opentelemetry-operator-and-collectors.md). This was done because these receivers fetch their metrics directly from the node.

The `k8scluster` receiver was added to the otel-collector deployment that runs in the shoot's control plane - this receiver talks directly to the `kube-apiserver`.

The `shoot-prometheus` instance in the shoot's control plane was configured to scrape the new metrics from the otel-collector that also runs in the shoot's control plane.

The architecture diagram of how the receivers are wired is displayed below:

**Note** that to enable a particular receiver in the otel collector and test changes locally, it has to be added in https://github.com/gardener/opentelemetry-collector/blob/main/manifest.yml and then an image has to be built from https://github.com/gardener/opentelemetry-collector

The image can then be loaded into the local gardener kind cluster by tagging and pushing it to the local docker registry:

$ docker tag europe-docker.pkg.dev/gardener-project/snapshots/gardener/otel/opentelemetry-collector:v0.0.2-dev-636ec4d-dirty localhost:5001/gardener-project/snapshots/gardener/otel/opentelemetry-collector:v0.0.2-dev-636ec4d-dirty

$ docker push localhost:5001/gardener-project/snapshots/gardener/otel/opentelemetry-collector:v0.0.2-dev-636ec4d-dirty

Finally, the image reference has to be changed in https://github.com/vitanovs/gardener/blob/f209fab82030e851b0151d7ef8b6421457585999/imagevector/containers.yaml#L190-L193

## Findings

Generally, the otel-collector receivers seem like a good approach due to the ability to process metrics directly in the pipelines defined in the otel-collector's configuration.
Additionally, in the future it will be easier to wire these metrics to different otel-collector instances, even ones provided by shoot owners.

However, not all metrics provided by the old prometheus exporters are covered by the otel receivers. Below follows a more detailed view on each receiver and its shortcomings.

A good point to note is that the configuration of the ingress for the otel-collector running in the shoot's control plane had to be extended to also process traffic received for the `/opentelemetry.proto.collector.metrics.v1.MetricsService/Export` path. Refs:
- https://github.com/vitanovs/gardener/blob/f209fab82030e851b0151d7ef8b6421457585999/pkg/component/observability/opentelemetry/collector/collector.go#L677-L693
- https://github.com/vitanovs/gardener/blob/f209fab82030e851b0151d7ef8b6421457585999/pkg/component/observability/opentelemetry/collector/collector.go#L758

### Hostmetrics

The `hostmetrics` receiver covers all of the metrics provided by the `node-exporter` with two exceptions:
1. There is no metric to indicate the time that it took for the node to boot, however there is a metric that specifies the time that has elapsed since the node has booted:
- `node_exporter`
- `node_boot_time` - the exact _timestamp_ of the Node's boot time ( for example `1765154682`)
- `hostmetrics`
- `system_uptime_seconds` - seconds __elapsed__ since the Node's boot time ( for example `136585` )
- The `node_boot_time` could be easily calculated with the `system_uptime_seconds`, although with reduced precision: `node_boot_time` = `now` - `system_uptime_seconds`
3. There was no obvious way on how to add support for the `textfile` collector used by `node-exporter`. The `textfile` collector is currently used to read metrics from `*.prom` files located under `/var/lib/node-exporter/textfile-collector`

There are also a few differences in the _names_ and _labels_ of metrics. Here is an [article](https://grafana.com/grafana/dashboards/20376-opentelemetry-collector-hostmetrics-node-exporter) from Grafana.

**Configuration** of the `hostmetrics` receiver for the otel-collector service that runs on nodes by editing the `pkg/component/extensions/operatingsystemconfig/original/components/opentelemetrycollector/templates/opentelemetry-collector-config.yaml.tpl` config file template:
1. Add the following entry to the `receivers` section

    receivers:
      hostmetrics:
        collection_interval: 10s
        scrapers:
          cpu:
          disk:
          load:
          filesystem:
          memory:
          network:
          system:

The `scrapers` section determines which metrics are scraped
2. Define a pipeline that processes `hostmetrics` by adding the following configuration to the `pipelines` section

    pipelines:
      metrics:
        receivers: [hostmetrics]
        processors: [resource/journal, batch]
        exporters: [otlp, debug]

### Kubeletstats

The `kubeletstats` receiver covers all metrics provided by `cAdvisor` with the exception of:
- `container_fs_reads_bytes_total`
- `container_fs_writes_bytes_total`

**Configuration** of the `kubeletstats` receiver for the otel-collector service that runs on nodes by editing the `pkg/component/extensions/operatingsystemconfig/original/components/opentelemetrycollector/templates/opentelemetry-collector-config.yaml.tpl` config file template:
1. Add the following entry to the `receivers` section

    receivers:
      kubeletstats:
        collection_interval: 20s
        initial_delay: 1s
        auth_type: "tls"
        ca_file: {{ .kubeletCA }}
        key_file: {{ .kubeletCertKey }}
        cert_file: {{ .kubeletCert }}
        endpoint: "https://127.0.0.1:10250"
        insecure_skip_verify: true

The `ca_file`, `key_file`, and `cert_file` contain the client certificates required to authenticate to the `kubelet`. They can be generated and provided via the OSC. Corresponding RBAC also have to be created to allow the otel-collector user (from the client certificate) to get the `/stats/summary` endpoint of the kubelet.
2. Extend the pipeline that processes `hostmetrics` to also process `kubeletstats` metrics

    pipelines:
      metrics:
        receivers: [hostmetrics, kubeletstats]
        processors: [resource/journal, batch]
        exporters: [otlp, debug]

3. Reuse or generate the secret that is used to hold the certificates for talking to the kubelet. During the hackathon we decided to reuse the already existing secret - `kube-apiserver-kubelet`, but for productive usage, a new secret should be generated.
4. Deploy the data entries from the `kube-apiserver-kubelet` secret to the node via the operating system config component that is used to deploy the otel-collector service on the node: https://github.com/vitanovs/gardener/blob/f209fab82030e851b0151d7ef8b6421457585999/pkg/component/extensions/operatingsystemconfig/original/components/opentelemetrycollector/component.go#L106-L133

### K8scluster
The `k8scluster` receiver covers very different metrics compared to the ones provided by `kube-state-metrics`. Additionally, it does not support receiving metrics for custom resources, e.g. `VerticalPodAutoscaler`.
It either has to be extended further, or we have to contribute our own receiver that fully replaces `kube-state-metrics`.

**Configuration** of the `k8scluster` receiver via the otel operator:
1. The `k8scluster` receiver requires access to the shoot's kube-apiserver. For this a "shoot access secret" has to be created for it with `gardenerutils.NewShootAccessSecret`.
2. Create the corresponding RBAC resources for the newly create "shoot access secret". For the purposes of the hackathon we have allowed access to everything, but this should be limited for productive cases.
3. Add a volume and volume mount to the opentelemetry collector operator resource for a generic kubeconfig using the `gardenerutils.GenerateGenericKubeconfigVolume` function
4. Add an environment `KUBECONFIG` that points to the generic kubeconfig to the opentelemetry collector operator resource. The file that this enviornment points to will be automatically used by the `k8scluster` resource