# Tobs (The Observability Stack for Kubernetes) ## Introduction In the world of computing observability has become mandatory to understand the state of infrastructure and applications. In the recent past Kubernetes has become the de facto solution for running the applications. The world of observability has multiple dimensions such as metrics, logs & traces. Metrics helps us to understand the state of application or resource. Logs provide the application level information to understand the behaviour of a particular action. Traces gives the insights into application flow and the bottlenecks in the business logic. Getting Observability from the Kubernetes cluster is definitely challenging as it involves installing multiple services & configuring them accordingly. **tobs is a one stop shop for all your kubernetes monitoring needs.** ## Diving into tobs Tobs is CLI tool designed to install all the observability stack you need for monitoring in your Kubernetes cluster and provides complete life cycle support on your monitoring stack. It abstracts all the actions for observability stack with a single cmd. Installing the monitoring stack is as simple as running: ``` tobs install ``` This will install ***Grafana, TimescaleDB, Promscale, Prometheus-Node-Exporter, Prometheus, Promlens and Kube-State-Metrics.*** All the components deployed will be configured to connect with other components. Also, Kubernetes dashbaords are pre-configured in grafana UI. You can visualise all the observability you need from your k8s cluster. ![](https://i.imgur.com/oM2KLhG.gif) ## Components ### TimescaleDB TimescaleDB is battle tested peta-byte scale relational time-series database. tobs by default deploys the timescaleDB to store all monitoring data for long term storage and grafana config data. You can find more details on timescaleDB here https://github.com/timescale/timescaledb ### Promscale Promscale is a prometheus adaptor to store & retrieve metrics in time-series database through through prometheus remote-read & remote-write api's. You can find more details on promscale here https://github.com/timescale/promscale ### Node-Exporter Node-Exporter is deployed to export all nodes related metrics from the kubernetes cluster. You can find more details on node-exporter here https://github.com/prometheus/node_exporter ### Prometheus Prometheus is an open-source systems monitoring and altering stack. It allows you to understand how your infrastructure & applications are performing from the metrics discovered by prometheus. You can find more details on prometheus here https://github.com/prometheus/prometheus ### Kube-State-Metrics Kube-state-metrics gives you overall information on kubernetes resources i.e. the status & count of k8s resources with visibility over the desired resources vs the current resources and the trend in your cluster over the period of time. You can find more details on kube-state-metrics here https://github.com/kubernetes/kube-state-metrics ### Promlens Tool to build promql queries with ease. Promlens offers possible functions per metric and helps you to build complex promql queries. You can find more details on promlens here https://promlens.com/ ### Grafana Grafana is visualisation tool to view rich dashboards based on the metrics scrpared from different targets. Deploying grafana using tobs will create rich dashboards out of the box that are required to monitor k8s cluster. You can find more details on grafana here https://github.com/grafana/grafana ## Usage tobs CLI installation `curl --proto '=https' --tlsv1.2 -sSLf https://tsdb.co/install-tobs-sh |sh` Available Commands: ``` $ tobs The Observability Stack is a tool that uses TimescaleDB as a compressed long-term store for time series metrics from Prometheus. This application is a CLI tool that allows users to quickly access the different components of Observability. Usage: tobs [command] Available Commands: grafana Subcommand for Grafana operations helm Subcommand for Helm operations help Help about any command install Alias for helm install metrics Subcommand for metrics operations port-forward Port-forwards TimescaleDB, Grafana, and Prometheus to localhost prometheus Subcommand for Prometheus operations promlens Subcommand for Grafana operations timescaledb Subcommand for TimescaleDB operations uninstall Alias for helm uninstall Flags: --config string config file (default is $HOME/.tobs.yaml) -h, --help help for tobs -n, --name string Helm release name (default "tobs") --namespace string Kubernetes namespace (default "default") Use "tobs [command] --help" for more information about a command. ``` ### Deploying monitoring stack `tobs install` This will successfully deploy all the above listed components to your k8s cluster. You can use `--name` flag to set helm release name default name is `tobs`, `--namespace` flag to deploy observability stack in namespace of your choice default is set to `default` namespace, `--chart-reference` flag to use helm chart of your choice default is set to `timescale/tobs`, `--filename` to use values.yml file of your choice with helm chart. ### Actions for Grafana: tobs CLI offers a grafana cmd to interact with grafana with ease. Actions such as get grafana instance password, change grafana instance password and port forwarding to locally interact with grafana. Usually to perform these actions we need view grafana secret & decode it from base64 and changing password needs secret updation. All this actions will be performed by tobs. #### get grafana password: ``` tobs grafana get-password ``` #### change grafana password: ``` tobs grafana change-password <PASSWORD> ``` #### port forward grafana By default this will port-forward to port `8080` of your local machine. Use `--port` flag to port-forward to port of your choice. ``` tobs grafana port-forward ``` Acessing `localhost:8080` will have pre-built dashbaords for kubernetes monitoring. i.e <img src="https://i.imgur.com/x81brpV.png" width="800" height="350" /> ### Actions for Helm: tobs CLI internally uses helm to deploy observability stack. You can also use helm cmds such helm install, helm uninstall, show-values i.e values.yml file and delete-data to delete persistent volume claims. #### install tobs using helm `tobs install` is an alias for below cmd ``` tobs helm install ``` #### uninstall tobs using helm `tobs uninstall` is an alias for below cmd ``` tobs helm uninstall ``` #### deletes all persistent volume claims ``` tobs helm delete-data ``` #### show values.yml file used in helm This cmd helps us to understand values passed over to helm chart for different components. ``` tobs helm show-values ``` ### Port-forward This helps to port-forward `TimescaleDB`, `Grafana`, `Prometheus`, `Promscale` and `Promlens` to localhost i.e TimescaleDB to `5432` Promlens to `8081` Grafana to `8080` Prometheus to `9090` Promscale to `9201` You can also port-forward to ports of your choice by using the flags exposed for different components. ``` tobs port-forward ``` ### Metrics You can perform metric specifc actions such as ```chunk-interval``` & ```retantion``` #### chunk-interval Shows the chunk-interval per metric i.e hybertable distribution per metric in TimescaleDB. ``` tobs metrics chunk-interval get go_goroutines ``` Set chunk-interval to a specifc metric ``` tobs metrics chunk-interval set go_threads 2h ``` Reset chunk-interval to a specifc metric to default ``` tobs metrics chunk-interval reset go_threads ``` Set default chunk-interval to all metrics except for which we set chunk-interval explicitly. ``` tobs metrics chunk-interval set-default 3h ``` #### retention Using TimescaleDB for long term storage of metrics allows you to set retention per metric. The default unit of metric retention is mentioned in days. Show the retention set per metric ``` tobs metrics retention get go_goroutines ``` Set retention to a specifc metric ``` tobs metrics retention set go_threads 2 ``` Reset retention to a specifc metric to default ``` tobs metrics retention reset go_threads ``` Set default retention to all metrics except for which we set retention explicitly. ``` tobs metrics retention set-default 3 ``` ### Actions for Prometheus This helps to port-forward prometheus to localhost i.e to port `9090` by default. Use `--port` flag to port-forward to port of your choice. #### port forward prometheus ``` tobs prometheus port-forward ``` ### Actions for Promlens This helps to port-forward promlens and promscale to localhost i.e promlens to `8081` and promscale to `9201` by default. Use `--port` flag to port-forward promlens and `--port-connector` flag to port-forward promscale to port of your choice. #### port forward promlens ``` tobs promlens port-forward ``` ### Actions for TimescaleDB This helps to perform timescaleDB operations such as get password for specific user using `--user` flag, change password for specific database and user using `--dbname` & `--user` flags, connect to psql prompt, and port-forward to localhost. By default tobs CLI uses DB & USER as `postgres` #### get password Use `--user` flag to get password for specifc user. ``` tobs timescaledb get-password ``` #### change password Use `--user` & `--dbname` flags to change password for specific user and database. ``` tobs timescaledb change-password ``` #### connect to psql prompt Opens up an interactive psql shell to query timescaledb. Use `master` flag to connect with master node and `--user` flag to connect to db using specifc user. ``` tobs timescaledb connect ``` #### port forward timescaledb By default this will port-forward to port `5432` of your local machine. Use `--port` flag to port-forward to port of your choice. ``` tobs timescaledb port-forward ``` ### Uninstall tobs To uninstall monitoring stack deployed by tobs. Use `--delete-data` flag to delete persistent volume claims (pvc's) and `--pvc` to remove pvc's. ``` tobs unintall ``` <br> Cheers!