---
# System prepended metadata

title: Tobs (The Observability Stack for Kubernetes)

---

# Tobs (The Observability Stack for Kubernetes)

## Introduction

In the world of computing observability has become mandatory to understand the state of infrastructure and applications. In the recent past Kubernetes has become the de facto solution for running the applications. The world of observability has multiple dimensions such as metrics, logs & traces. Metrics helps us to understand the state of application or resource. Logs provide the application level information to understand the behaviour of a particular action. Traces gives the insights into application flow and the bottlenecks in the business logic.

Getting Observability from the Kubernetes cluster is definitely challenging as it involves installing multiple services & configuring them accordingly. 

**tobs is a one stop shop for all your kubernetes monitoring needs.**

## Diving into tobs

Tobs is CLI tool designed to install all the observability stack you need for monitoring in your Kubernetes cluster and provides complete life cycle support on your monitoring stack. It abstracts all the actions for observability stack with a single cmd.

Installing the monitoring stack is as simple as running:

```
tobs install
```

This will install ***Grafana, TimescaleDB, Promscale, Prometheus-Node-Exporter, Prometheus, Promlens and Kube-State-Metrics.***

All the components deployed will be configured to connect with other components. Also, Kubernetes dashbaords are pre-configured in grafana UI. You can visualise all the observability you need from your k8s cluster.

![](https://i.imgur.com/oM2KLhG.gif)




## Components

### TimescaleDB

TimescaleDB is battle tested peta-byte scale relational time-series database. tobs by default deploys the timescaleDB to store all monitoring data for long term storage and grafana config data. 
You can find more details on timescaleDB here https://github.com/timescale/timescaledb

### Promscale

Promscale is a prometheus adaptor to store & retrieve metrics in time-series database through through prometheus remote-read & remote-write api's.
You can find more details on promscale here https://github.com/timescale/promscale

### Node-Exporter

Node-Exporter is deployed to export all nodes related metrics from the kubernetes cluster.
You can find more details on node-exporter here https://github.com/prometheus/node_exporter

### Prometheus

Prometheus is an open-source systems monitoring and altering stack. It allows you to understand how your infrastructure & applications are performing from the metrics discovered by prometheus.
You can find more details on prometheus here https://github.com/prometheus/prometheus


### Kube-State-Metrics

Kube-state-metrics gives you overall information on kubernetes resources i.e. the status & count of k8s resources with visibility over the desired resources vs the current resources and the trend in your cluster over the period of time.
You can find more details on kube-state-metrics here https://github.com/kubernetes/kube-state-metrics

### Promlens

Tool to build promql queries with ease. Promlens offers possible functions per metric and helps you to build complex promql queries.
You can find more details on promlens here https://promlens.com/

### Grafana

Grafana is visualisation tool to view rich dashboards based on the metrics scrpared from different targets. Deploying grafana using tobs will create rich dashboards out of the box that are required to monitor k8s cluster.
You can find more details on grafana here https://github.com/grafana/grafana

## Usage 

tobs CLI installation

`curl --proto '=https' --tlsv1.2 -sSLf  https://tsdb.co/install-tobs-sh |sh`

Available Commands:

```
$ tobs
The Observability Stack is a tool that uses TimescaleDB as a 
compressed long-term store for time series metrics from Prometheus. This
application is a CLI tool that allows users to quickly access the different
components of Observability.

Usage:
  tobs [command]

Available Commands:
  grafana      Subcommand for Grafana operations
  helm         Subcommand for Helm operations
  help         Help about any command
  install      Alias for helm install
  metrics      Subcommand for metrics operations
  port-forward Port-forwards TimescaleDB, Grafana, and Prometheus to localhost
  prometheus   Subcommand for Prometheus operations
  promlens     Subcommand for Grafana operations
  timescaledb  Subcommand for TimescaleDB operations
  uninstall    Alias for helm uninstall

Flags:
      --config string      config file (default is $HOME/.tobs.yaml)
  -h, --help               help for tobs
  -n, --name string        Helm release name (default "tobs")
      --namespace string   Kubernetes namespace (default "default")

Use "tobs [command] --help" for more information about a command.
```

### Deploying monitoring stack

`tobs install`

This will successfully deploy all the above listed components to your k8s cluster.

You can use `--name` flag to set helm release name default name is `tobs`, `--namespace` flag to deploy observability stack in namespace of your choice default is set to `default` namespace, `--chart-reference` flag to use helm chart of your choice default is set to `timescale/tobs`, `--filename` to use values.yml file of your choice with helm chart.

### Actions for Grafana:

tobs CLI offers a grafana cmd to interact with grafana with ease. Actions such as get grafana instance password, change grafana instance password and port forwarding to locally interact with grafana. Usually to perform these actions we need view grafana secret & decode it from base64 and changing password needs secret updation. All this actions will be performed by tobs.

#### get grafana password:

```
tobs grafana get-password
```

#### change grafana password:

```
tobs grafana change-password <PASSWORD>
```

#### port forward grafana

By default this will port-forward to port `8080` of your local machine. Use `--port` flag to port-forward to port of your choice. 

```
tobs grafana port-forward
```

Acessing `localhost:8080` will have pre-built dashbaords for kubernetes monitoring. i.e



<img src="https://i.imgur.com/x81brpV.png" width="800" height="350" />

### Actions for Helm:

tobs CLI internally uses helm to deploy observability stack. You can also use helm cmds such helm install, helm uninstall, show-values i.e values.yml file and delete-data to delete persistent volume claims.

#### install tobs using helm 

`tobs install` is an alias for below cmd

```
tobs helm install
```

#### uninstall tobs using helm


`tobs uninstall` is an alias for below cmd

```
tobs helm uninstall
```

#### deletes all persistent volume claims

```
tobs helm delete-data 
```

#### show values.yml file used in helm

This cmd helps us to understand values passed over to helm chart for different components.

```
tobs helm show-values
```

### Port-forward

This helps to port-forward `TimescaleDB`, `Grafana`, `Prometheus`, `Promscale` and `Promlens` to localhost i.e 

TimescaleDB to `5432`
Promlens to `8081`
Grafana to `8080`
Prometheus to `9090`
Promscale to `9201`

You can also port-forward to ports of your choice by using the flags exposed for different components. 

```
tobs port-forward
```

### Metrics

You can perform metric specifc actions such as ```chunk-interval``` & ```retantion``` 

#### chunk-interval

Shows the chunk-interval per metric i.e hybertable distribution per metric in TimescaleDB.

```
tobs metrics chunk-interval get go_goroutines
```

Set chunk-interval to a specifc metric

```
tobs metrics chunk-interval set go_threads 2h
```

Reset chunk-interval to a specifc metric to default

```
tobs metrics chunk-interval reset go_threads
```

Set default chunk-interval to all metrics except for which we set chunk-interval explicitly.

```
tobs metrics chunk-interval set-default 3h
```

#### retention

Using TimescaleDB for long term storage of metrics allows you to set retention per metric.

The default unit of metric retention is mentioned in days.

Show the retention set per metric
```
tobs metrics retention get go_goroutines
```

Set retention to a specifc metric

```
tobs metrics retention set go_threads 2
```

Reset retention to a specifc metric to default

```
tobs metrics retention reset go_threads
```

Set default retention to all metrics except for which we set retention explicitly.

```
tobs metrics retention set-default 3
```

### Actions for Prometheus

This helps to port-forward prometheus to localhost i.e to port `9090` by default. Use `--port` flag to port-forward to port of your choice. 

#### port forward prometheus

```
tobs prometheus port-forward
```

### Actions for Promlens

This helps to port-forward promlens and promscale to localhost i.e 
promlens to `8081` and promscale to `9201` by default. Use `--port` flag to port-forward promlens and `--port-connector` flag to port-forward promscale to port of your choice. 

#### port forward promlens

```
tobs promlens port-forward
```

### Actions for TimescaleDB

This helps to perform timescaleDB operations such as get password for specific user using `--user` flag, change password for specific database and user using `--dbname` & `--user` flags, connect to psql prompt, and port-forward to localhost.

By default tobs CLI uses DB & USER as `postgres`

#### get password

Use `--user` flag to get password for specifc user.

```
tobs timescaledb get-password
```

#### change password

Use `--user` & `--dbname` flags to change password for specific user and database.

```
tobs timescaledb change-password
```

#### connect to psql prompt

Opens up an interactive psql shell to query timescaledb.

Use `master` flag to connect with master node and `--user` flag to connect to db using specifc user.

```
tobs timescaledb connect
```

#### port forward timescaledb

By default this will port-forward to port `5432` of your local machine. Use `--port` flag to port-forward to port of your choice. 

```
tobs timescaledb port-forward
```

### Uninstall tobs

To uninstall monitoring stack deployed by tobs. Use `--delete-data` flag to delete persistent volume claims (pvc's) and `--pvc` to remove pvc's.

```
tobs unintall
```

<br>


Cheers!