<style type="text/css"> .reveal p { font-size: 28px; text-align: left; } .reveal ul { font-size: 28px; display: block; } .reveal ol { font-size: 28px; display: block; } .reveal h1 { font-size: 80px; display: block; } .reveal h1 { font-size: 72px; display: block; } .reveal h2 { font-size: 64px; display: block; } .reveal h3 { font-size: 56px; display: block; } .reveal h4 { font-size: 48px; display: block; } .reveal h5 { font-size: 40px; display: block; } .reveal h6 { font-size: 32px; display: block; } </style> ## <b style="color:darkgrey;text-align:center">Prometheus Overview</b> #### <b style="color:lightgrey;text-align:center"> B06902031 資工四 何承勳</b> --- ## <b style="color:darkgrey">Outline</b> <font color=gray> - Introduction - Architecture - Components - Prometheus Server - Exporters - Push Gateway - Alertmanager - Consoles and Dashboards - Service Discovery - Pros and Cons </font> --- ## <b style="color:darkgrey">Outline</b> <font color=gray> - <b style="color: orange">Introduction</b> - Architecture - Components - Prometheus Server - Exporters - Push Gateway - Alertmanager - Consoles and Dashboards - Service Discovery - Pros and Cons </font> ---- ## <b style="color: orange;">Introduction</b> - Prometheus is an pen-source systems monitoring and alerting toolkit. It is written in <b style="color: deepskyblue">Go</b>, licensed under the <b style="color: deepskyblue"> Apache 2 License</b> - Prometheus collects metrics from configured targets at given intervals, and store the time series data in its database. - Prometheus runs rules over the collected data to aggregate these data or generate alerts. - Several dashboards are available for administrator to visualize the collected data. ---- ## <b style="color: orange;">Features</b> - Prometheus stores all data as time series, which can be identified by <b style="color: deepskyblue">metric names</b>. - Prometheus provides a query language called <b style="color: deepskyblue">PromQL</b> that allows the user to query and aggregate time series data. - Prometheus collects data via <b style="color: deepskyblue">HTTP PULL method</b>. Alternatively, pushing mechanism is supported through <b style="color: deepskyblue">push gateway</b>. - Prometheus can trigger alerts if certain condition is observed to be true. --- ## <b style="color:darkgrey">Outline</b> <font color=gray> - Introduction - <b style="color: orange;">Architecture</b> - Components - Prometheus Server - Exporters - Push Gateway - Alertmanager - Consoles and Dashboards - Service Discovery - Pros and Cons </font> ----  --- ## <b style="color:darkgrey">Outline</b> <font color=gray> - Introduction - Architecture - <b style="color: orange;">Components</b> - Prometheus Server - Exporters - Push Gateway - Alertmanager - Consoles and Dashboards - Service Discovery </font> --- ## <b style="color:darkgrey">Outline</b> <font color=gray> - Introduction - Architecture - Components - <b style="color: orange;">Prometheus Server</b> - Exporters - Push Gateway - Alertmanager - Consoles and Dashboards - Service Discovery - Pros and Cons </font> ---- ## <b style="color: orange;">Prometheus Server</b> - The Prometheus Server retrieves data from monitored target, stores time series data into the database, and provide interface for users to query the database. - Generically, consists of three components: - <b style="color: deepskyblue">Time Series Database (TSDB)</b> - <b style="color: deepskyblue">HTTP Server</b> - <b style="color: deepskyblue">Prometheus Quering Language (PromQL)</b> ---- ## <b style="color: orange">TSDB</b> - Prometheus server consists of a Time Series database (TSDB). A <b style="color: deepskyblue">TSDB</b> is a database optimized for handling time series data. - Prometheus stores all data by time series. Every time series is uniquely identified by its <b style="color: deepskyblue">metric name</b> and optional key-value pairs called <b style="color: deepskyblue">labels</b>. ---- ### <b style="color: orange">Metrics</b> - For example, a time series with the metric name <span style="color:limegreen;font-family:monospace">prometheus_http_requests_total</span> (which indicates the number of accumulated http requests to Prometheus Server), and the labels <span style="color:limegreen;font-family:monospace">method="POST"</span> (which specifies the number of POST requests) could be written like the following: ``` prometheus_http_requests_total{method="POST", handler="/messages"} ``` ---- ### <b style="color: orange">Metrics</b> - Metrics can be categorized into four types: 1. <b style="color: deepskyblue">Counter</b>: Metrics that can be accumulated, such as the number of occurrences of an HTTP Get requests. 2. <b style="color: deepskyblue">Gauge</b>: Any change metric that is instantaneous and independent of time, such as memory usage. 3. <b style="color: deepskyblue">Histogram</b>: Mainly used to represent data sampling within a period of time. 4. <b style="color: deepskyblue">Summary</b>: Similar to Histogram, it is used to represent the summary of data sampling in a time range. ---- ### <b style="color: orange">PromQL</b> - <b style="color:deepskyblue">PromQL</b> (Prometheus Query Language) is a quering language provided by Prometheus that allows the user to select, examine, and aggregate time series data. - [Official Documentation of PromQL](https://prometheus.io/docs/prometheus/latest/querying/examples/) ---- ### <b style="color: orange">HTTP Server</b> - Prometheus server provides a <b style="color:deepskyblue">HTTP API</b> for users to query the database. - The current stable HTTP API is reachable under <span style="color:limegreen;font-family:monospace">/api/v1</span> on a Prometheus server. - The API response format is in <b style="color:deepskyblue">JSON</b>. ---- ### <b style="color: orange">HTTP Server</b> - Example: We can use curl to query the Prometheus server: ```bash curl 'http://localhost:9090/api/v1/query?query=up ``` - The Prometheus server will then return the result in JSON format: ```json { "status" : "success", "data" : { "resultType" : "vector", "result" : [ { "metric" : { "__name__" : "up", "job" : "prometheus", "instance" : "localhost:9090" }, "value": [ 1435781451.781, "1" ] }, ] } } ``` --- ## <b style="color:darkgrey">Outline</b> <font color=gray> - Introduction - Architecture - Components - Prometheus Server - <b style="color: orange;">Exporters</b> - Push Gateway - Alertmanager - Consoles and Dashboards - Service Discovery - Pros and Cons </font> ---- ## <b style="color: orange">Exporters</b> - <b style="color:deepskyblue">Exporters</b> are used to expose metrics of third-party services to Prometheus Server. The Exporters are installed on the monitored device. - Exporters will expose an <b style="color:deepskyblue">http endpoint</b> for Prometheus server to retrieve metrics. Prometheus mainly uses <b style="color:deepskyblue">HTTP PULL</b> method to collect metrices. It retrieves metrics by periodically pulling metrics from the monitored target's http endpoints. - Exporters is written using <b style="color:deepskyblue">Prometheus Client Libraries</b>. The library supports many differnt languages. The client library provides an API that can sends the metrics back to the server when Prometheus scrapes the target's HTTP endpoint. ---- ### <b style="color: orange">Exporter</b> - <b style="color:deepskyblue">Node exporter</b> is one of the most common official exporter. It exposes some hardware and OS metrics of UNIX kernels. For example: CPU usage, memory statics, disk I/O statistics, network statistics, and so on. ([Node Exporter Github Page](https://github.com/prometheus/node_exporter)) - <b style="color:deepskyblue">Mysql server exporter</b> is another common official exporter. It allows us to monitor, measure database performance, examine resource utilization, and so on. ([MySQL Exporter Github Page](https://github.com/prometheus/mysqld_exporter)) - If no existing exporters meet our need, we can write our own exporter using <b style="color:deepskyblue">Prometheus Client Libraries</b>. --- ## <b style="color:darkgrey">Outline</b> <font color=gray> - Introduction - Architecture - Components - Prometheus Server - Exporters - <b style="color: orange;">Push Gateway</b> - Alertmanager - Consoles and Dashboards - Service Discovery - Pros and Cons </font> ---- ### <b style="color: orange">Pushgateway</b> - Occasionally, we might need to monitor components which cannot be scraped. In this case, the <b style="color:deepskyblue">Pushgateway</b> is used to tackle the problem. These metrices will be pushed onto the Pushgateway first, then Prometheus will periodically pull the metrics from the Pushgateway. - In the official documentation, it states that "*Usually, the only valid use case for the Pushgateway is for capturing the outcome of a service-level batch job*". An example of "service-level" batch job is deleting a number of users for an entire service. is a discrete job which is not related to a specific machine. - In conclusion, the Pushgateway is seldomly used. --- ## <b style="color:darkgrey">Outline</b> <font color=gray> - Introduction - Architecture - Components - Prometheus Server - Exporters - Push Gateway - <b style="color: orange;">Alertmanager</b> - Consoles and Dashboards - Service Discovery - Pros and Cons </font> ---- ### <b style="color: orange">Alert Manager</b> - By defining alarm rule in Prometheus' configuration file, Prometheus will periodically calculate the alarm rule. If it meets the alarm trigger conditions, it will push an alarm to the <b style="color: deepskyblue;">Alertmanager</b>. - The Alertmanager can further inform the administrator some abnormal events via email, Pagerduty, etc. --- ## <b style="color:darkgrey">Outline</b> <font color=gray> - Introduction - Architecture - Components - Prometheus Server - Exporters - Push Gateway - Alertmanager - <b style="color: orange;">Consoles and Dashboards</b> - Service Discovery - Pros and Cons </font> ---- ### <b style="color: orange">Expression Browser</b> - The expression browser is available at <span style="color:limegreen;font-family:monospace">/graph</span> on the Prometheus server. It allowing us to enter any PromQL query and see its result in a table or a graph. ---- ### <b style="color: orange">Grafana</b> - <b style="color: deepskyblue">Grafana</b> is a universal visualization tool suitable for visualizing and displaying data stored in different databases including Prometheus. ---- ### <b style="color: orange">Console Template</b> - Prometheus consists of a simple built-in console template that allows users to create any console interface. - [Official documentation of Prometheus Console Template](https://prometheus.io/docs/visualization/consoles/) --- ## <b style="color:darkgrey">Outline</b> <font color=gray> - Introduction - Architecture - Components - Prometheus Server - Exporters - Push Gateway - Alertmanager - Consoles and Dashboards - <b style="color: orange;">Service Discovery</b> - Pros and Cons ---- ### <b style="color: orange">Service Discovery</b> - In cloud environment, there is no fixed monitoring target, and nearly every monitored object in the cloud changes dynamically. Thus, we cannot statically monitor every device in the cloud. - The solution to the above problem is introducing an intermediate agent. This agent has access to all current monitored targets. - Prometheus only needs to ask the agent what monitoring targets there are. Such mechanism is called <b style="color: deepskyblue;">service discovery</b>. ---- ### <b style="color: orange">Example</b> - In some cloud environments like <b style="color: deepskyblue">AWS</b>, Prometheus has the ability to find all cloud hosts that need to be monitored by using the API provided by the platform. - In <b style="color: deepskyblue">Kubernetes</b>, The master node manages all nodes information, Thus, Prometheus only need to interact with the master node to find all the containers and service objects that need to be monitored. --- - Introduction - Architecture - Components - Prometheus Server - Exporters - Push Gateway - Alertmanager - Consoles and Dashboards - Service Discovery - <b style="color: orange;">Pros and Cons</b> ---- ### <b style="color: orange">Pros</b> - Prometheus is mainly used for event monitoring and event alerting. It works prticularly well for recording purely time series data. - Prometheus fits well in monitoring dynamic service-oriented cloud environments such as Kubernetes. - Prometheus has higher reliability since Prometheus server is a standalone monitoring system, ane it does not depending on network storage or other remote services. ---- ### <b style="color: orange">Cons</b> - Prometheus does not offer durable long-term storage. The data storage of Prometheus is ephemeral since is mainly used for event monitoring and alerting. - Prometheus does not support logging. Prometheus is designed to collect and process metrics, not an event logging system. ---- ### <b style="color: orange">Concusion</b> - In our project, Elastic Stack can be used to perform long-term data storage, monitoring, and data retrieval, while Prometheus can be used to perform short-term event monitoring and alerting. - Since Prometheus works well in monitoring cloud enviroment, it can be deployed into our Kubernetes and perform monitoring on the entire opKubernetes.