## BLOG26: Prometheus Metric collection flow --- # 🧩 **What type of resource is Prometheus Adapter?** ### βœ”οΈ **Prometheus Adapter is a Deployment** Installed via Helm β†’ creates: * **Deployment** (pods running the adapter) * **Service** * **APIService** (`custom.metrics.k8s.io/v1beta1`) * **ConfigMap** (rules) * **ClusterRole / ClusterRoleBinding** * **ServiceAccount** πŸ’‘ **Important:** Prometheus Adapter registers itself as a **Kubernetes API extension**: ``` custom.metrics.k8s.io ``` But the core object running it is a **Deployment**. --- # 🌱 **What type of resource is PodMonitor?** ### βœ”οΈ **PodMonitor is a Custom Resource (CRD)** Created by the **Prometheus Operator (kube-prometheus-stack)**. CRD: ``` monitoring.coreos.com/v1 kind: PodMonitor ``` It tells Prometheus: > β€œScrape **these pods**, at **this path**, on **this port**.” It **does NOT** run anything β€” it’s **just configuration** that the Prometheus Operator converts into Prometheus scrape configs. --- # πŸš€ **Now the Whole Metric Exposure Flow β€” Brief, Beautiful & Complete** Below is the 10-second summary you can use in your slides or training: --- ## **πŸ”Ή Step 1 β€” Application exposes metrics** Your dummy app exposes: ``` /metrics β†’ myapp_queue_length ``` in Prometheus format. --- ## **πŸ”Ή Step 2 β€” PodMonitor tells Prometheus to scrape** `PodMonitor` selects the pods and creates the scrape config. Prometheus Operator loads the config β†’ Prometheus begins scraping. --- ## **πŸ”Ή Step 3 β€” Prometheus stores the metric** Prometheus stores: ``` myapp_queue_length{pod="dummy-queue-app-XYZ"} ``` --- ## **πŸ”Ή Step 4 β€” Prometheus Adapter translates Prometheus metrics β†’ K8s Custom Metrics API** Adapter rule converts: ``` Prometheus query β†’ Kubernetes API metric ``` Example internal query: ``` myapp_queue_length{namespace="queue-demo", pod="..."} ``` Adapter exposes it at: ``` /apis/custom.metrics.k8s.io/v1beta1/namespaces/queue-demo/pods/*/myapp_queue_length ``` Now Kubernetes can β€œsee” the metric. --- ## **πŸ”Ή Step 5 β€” HPA fetches metric from Custom Metrics API** HPA runs a loop: ``` GET custom.metrics.k8s.io/v1beta1/.../myapp_queue_length ``` Compares: * current queue length * desired target --- ## **πŸ”Ή Step 6 β€” HPA scales the Deployment** If queue > target: ``` scale up (replicas++) ``` If queue < target: ``` scale down (replicas--) ``` --- # πŸ—ΊοΈ **Full Metric Flow Diagram (ASCII)** Copy-paste friendly: ``` β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ Dummy App (Pods) β”‚ β”‚ exposes /metrics β”‚ β”‚ myapp_queue_length β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ PodMonitor (CRD) β”‚ β”‚ β€œscrape these pods” β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ generated config β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ Prometheus Operator β”‚ β”‚ creates scrape jobs β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ Prometheus β”‚ β”‚ stores metric values β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ PromQL query β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ Prometheus Adapter (Deploy) β”‚ β”‚ translates PromQL β†’ K8s API metric β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ exposes: β”‚ custom.metrics.k8s.io β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ Kubernetes Custom Metrics API (APIService) β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ HPA queries: β”‚ /apis/custom.metrics.k8s.io/…/myapp_queue_length β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ HPA (HorizontalPodAutoscaler) β”‚ decides scale up/down β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ Deployment replicas β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` --- # 🎁 **In One Line** **Pod β†’ PodMonitor β†’ Prometheus β†’ Prometheus Adapter β†’ Custom Metrics API β†’ HPA β†’ Scaling** ---