# Prometheus dan Grafana
## Preperation
- docker
- docker-compose
- git
- make
- github account
### Clone sample project
https://github.com/sofianhw/lumen-prometheus-sample
### Generate github token
https://github.com/settings/tokens
## Prometheus
### Data types
Prometheus uses four data types for metrics:
- scalar
- instant vector
- range vector
- string
The most fundamental data type of Prometheus is scalar, which represents a floating point value. Examples of scalars include 0, 18.12, and 1000000. All calculations in Prometheus are floating point operations.
When you group together scalars as a set of metrics in a single point in time, you get the instant vector data type. When you run a query asking for only the name of a metric, such as test_some_histogram_bucket, the response is an instant vector. Since metrics have both names and labels (which I’ll cover a little later), a single name may contain many values, and that is why it’s a vector rather than just a scalar.
An array of vectors over time gives you the range vector. Neither Grafana nor the built-in Prometheus expression browser makes graphs out of range vectors directly, but rather uses instant vectors or scalars independently calculated for different points in time.
Because of this, range vectors are typically wrapped with a function that transforms it into an instant vector (such as rate, delta, or increase) to be plotted. Syntactically, you get a range vector when you query an instant vector and append a time selector such as [5m]. The simplest range vector query is an instant vector with a time selector, such as test_some_histogram_bucket[1h].
### Simple time series selection
Return all time series with the metric http_requests_total:
> http_requests_total
Return all time series with the metric http_requests_total and the given job and handler labels:
> http_requests_total{type="blue"}
Return a whole range of time (in this case 5 minutes) for the same vector, making it a range vector:
> http_requests_total{type="blue"}[1m]
To select all HTTP status codes except 4xx ones, you could run:
> http_requests_total{status!~"4.."}
### Using functions, operators, etc
Return the per-second rate for all time series with the http_requests_total metric name, as measured over the last 5 minutes:
> rate(http_requests_total[5m])
Assuming that the http_requests_total time series all have the labels job (fanout by job name) and instance (fanout by instance of the job), we might want to sum over the rate of all instances, so we get fewer output time series, but still preserve the job dimension:
> sum by (job) (
rate(http_requests_total[5m])
)
If we have two different metrics with the same dimensional labels, we can apply binary operators to them and elements on both sides with the same label set will get matched and propagated to the output. For example, this expression returns the unused memory in MiB for every instance (on a fictional cluster scheduler exposing these metrics about the instances it runs):
> (instance_memory_limit_bytes - instance_memory_usage_bytes) / 1024 / 1024
we could get the top 3 CPU users grouped by application (app) and process type (proc) like this:
> topk(3, sum by (app, proc) (rate(instance_cpu_time_ns[5m])))
## Prometheus exporter untuk Laravel
### Overview
This application was created in order to be able to test various aspects and functionality of the prometheus exporter project. It may also act as a dockerized test-bed installation of Lumen.
This application using two library for implement Promotheus:
- Wrapper [arquivei/laravel-prometheus-exporter](https://github.com/arquivei/laravel-prometheus-exporter/)
- Client [promphp/prometheus_client_php](https://github.com/PromPHP/prometheus_client_php)
### Library
#### Wrapper
##### Installation
Add the repository to composer.json
```composer.json
"repositories": [
{
"type": "vcs",
"url": "https://github.com/arquivei/laravel-prometheus-exporter"
}
],
```
Install the package via composer
```bash
composer require arquivei/laravel-prometheus-exporter
```
After that you may enable facades and register the facade in your application's `bootstrap/app.php`
```php
$userAliases = [
// ...
Arquivei\LaravelPrometheusExporter\PrometheusFacade::class => 'Prometheus',
];
$app->withFacades(true, $userAliases);
```
Then you should register the service provider in `bootstrap/app.php`
```php
$app->register(Arquivei\LaravelPrometheusExporter\PrometheusServiceProvider::class);
```
Please see below for instructions on how to enable metrics on Application routes, Guzzle calls and SQL queries.
##### Configuration
The package has a default configuration which uses the following environment variables.
```
PROMETHEUS_NAMESPACE=app
PROMETHEUS_METRICS_ROUTE_ENABLED=true
PROMETHEUS_METRICS_ROUTE_PATH=metrics
PROMETHEUS_METRICS_ROUTE_MIDDLEWARE=null
PROMETHEUS_COLLECT_FULL_SQL_QUERY=true
PROMETHEUS_STORAGE_ADAPTER=memory
PROMETHEUS_REDIS_HOST=localhost
PROMETHEUS_REDIS_PORT=6379
PROMETHEUS_REDIS_TIMEOUT=0.1
PROMETHEUS_REDIS_READ_TIMEOUT=10
PROMETHEUS_REDIS_PERSISTENT_CONNECTIONS=0
PROMETHEUS_REDIS_PREFIX=PROMETHEUS_
```
To customize the configuration values you can either override the environment variables above (usually this is done in your application's `.env` file), or you can copy the included [prometheus.php](config/prometheus.php)
to `config/prometheus.php`, edit it and use it in your application as follows:
```php
$app->loadComponent('prometheus', [
Arquivei\LaravelPrometheusExporter\PrometheusServiceProvider::class
]);
```
##### Metrics
The package allows you to observe metrics on:
* Application routes. Metrics on request method, request path and status code.
* Guzzle calls. Metrics on request method, URI and status code.
* SQL queries. Metrics on SQL query and query type.
In order to observe metrics in application routes (the time between a request and response),
you should register the following middleware in your application's `bootstrap/app.php`:
```php
$app->middleware([
Arquivei\LaravelPrometheusExporter\RouteMiddleware::class,
]);
```
The labels exported are
```php
[
'method',
'route',
'status_code',
]
```
To observe Guzzle metrics, you should register the following provider in `bootstrap/app.php`:
```php
$app->register(Arquivei\LaravelPrometheusExporter\GuzzleServiceProvider::class);
```
The labels exported are
```php
[
'method',
'external_endpoint',
'status_code'
]
```
To observe SQL metrics, you should register the following provider in `bootstrap/app.php`:
```php
$app->register(Arquivei\LaravelPrometheusExporter\DatabaseServiceProvider::class);
```
The labels exported are
```php
[
'query',
'query_type',
]
```
Note: you can disable logging the full query by turning off the configuration of `PROMETHEUS_COLLECT_FULL_SQL_QUERY`.
#### Prometheus Client Php
##### Installation
Add as [Composer](https://getcomposer.org/) dependency:
```sh
composer require promphp/prometheus_client_php
```
##### Usage
A simple counter:
```php
\Prometheus\CollectorRegistry::getDefault()
->getOrRegisterCounter('', 'some_quick_counter', 'just a quick measurement')
->inc();
```
Write some enhanced metrics:
```php
$registry = \Prometheus\CollectorRegistry::getDefault();
$counter = $registry->getOrRegisterCounter('test', 'some_counter', 'it increases', ['type']);
$counter->incBy(3, ['blue']);
$gauge = $registry->getOrRegisterGauge('test', 'some_gauge', 'it sets', ['type']);
$gauge->set(2.5, ['blue']);
$histogram = $registry->getOrRegisterHistogram('test', 'some_histogram', 'it observes', ['type'], [0.1, 1, 2, 3.5, 4, 5, 6, 7, 8, 9]);
$histogram->observe(3.5, ['blue']);
$summary = $registry->getOrRegisterSummary('test', 'some_summary', 'it observes a sliding window', ['type'], 84600, [0.01, 0.05, 0.5, 0.95, 0.99]);
$histogram->observe(5, ['blue']);
```
Manually register and retrieve metrics (these steps are combined in the `getOrRegister...` methods):
```php
$registry = \Prometheus\CollectorRegistry::getDefault();
$counterA = $registry->registerCounter('test', 'some_counter', 'it increases', ['type']);
$counterA->incBy(3, ['blue']);
// once a metric is registered, it can be retrieved using e.g. getCounter:
$counterB = $registry->getCounter('test', 'some_counter')
$counterB->incBy(2, ['red']);
```
Expose the metrics:
```php
$registry = \Prometheus\CollectorRegistry::getDefault();
$renderer = new RenderTextFormat();
$result = $renderer->render($registry->getMetricFamilySamples());
header('Content-type: ' . RenderTextFormat::MIME_TYPE);
echo $result;
```
Change the Redis options (the example shows the defaults):
```php
\Prometheus\Storage\Redis::setDefaultOptions(
[
'host' => '127.0.0.1',
'port' => 6379,
'password' => null,
'timeout' => 0.1, // in seconds
'read_timeout' => '10', // in seconds
'persistent_connections' => false
]
);
```
Using the InMemory storage:
```php
$registry = new CollectorRegistry(new InMemory());
$counter = $registry->registerCounter('test', 'some_counter', 'it increases', ['type']);
$counter->incBy(3, ['blue']);
$renderer = new RenderTextFormat();
$result = $renderer->render($registry->getMetricFamilySamples());
```
### Installation
In order to run this application you'll need [Docker](https://docs.docker.com/install/)
along with [docker-compose](https://docs.docker.com/compose/install/).
For some functions you will also need cURL.
```bash=
git clone https://github.com/sofianhw/lumen-prometheus-sample.git
cd lumen-prometheus-sample
```
### Usage
The most commands needed exist in the `Makefile`. This includes
common commands for Docker, as well as commands for extra functionality,
like printing url of the available services to stdout.
#### Basic usage
- To run this project and test the results you simply need to run
```bash
$ make up
```
- head to `localhost:8080/test` (default port) make a few requests or use:
```bash
$ make call
```
`call` will perform a number of requests via curl. Default is 100 requests.
If you wish to change the number head to `.env.docker` and set `TIMES_TO_CALL_EXAMPLE_ROUTE`
to your preference.
- The last step is to open Grafana and see your metrics!
Grafana listens to `localhost:3326` (default port).
#### Useful commands
- To print all the available commands
```bash
$ make help
```
- To create and start the application services run:
```bash
$ make up
```
- To start the application services run:
```bash
$ make start
```
- To stop the application services run:
```bash
$ make stop
```
- To completely remove the application services run:
```bash
$ make down
```
#### Change application ports
The ports registered in the example application might be already
taken. If you happen to have conflicts you need to change the port having
conflicts in `.env.docker`.
E.g.
The Grafana port is registered at `3326`
```dotenv
EXAMPLE_APP_GRAFANA_PORT=3326
```
You'll need to change it as:
```dotenv
EXAMPLE_APP_GRAFANA_PORT=3111
```
#### Rebuild the example app container
To remove the built image and create it again:
```bash
$ make rebuild
```
#### Calling the route
The example route for testing is at `localhost:8080/test`
Each time you perform `make up` all the services get printed.
To print all the available services use:
```bash
$ make generate-links
```
#### Updating Grafana
Each time you perform `make up` the `dashboard.json` Grafana dashboards along with
the available data-sources, get updated.
To perform the update by yourself use:
```bash
$ make provision
```
The default Grafana credentials are:
```
username: admin
password: admin
```
To check the diagrams containing metrics, head to your
given Grafana URL and click on `Home -> App Overview`.
Prometheus scrapes every 30 seconds so be patient and
make enough calls to see your metrics.
An example view if you have completed the process successfully
should be:
Sign in with Wallet
Connect another wallet