# Part 3: Introduction to TIG Stacks ###### tags: `TA Stuff RP2` `Raspberry Pi Pico` The TIG (Telegraf, Influx and Grafana) Stack is an acronym for a platform of open source tools built to make collection, storage, graphing, and alerting on time series data easy. In this exercise we are going to set up a TIG-stack using Docker and Docker-compose for easy deployment. ![](https://i.imgur.com/TzgFn2G.png) A **time series** is simply any set of values with a timestamp where time is a meaningful component of the data. The classic real world example of a time series is stock currency exchange price data. When working with IoT-sensors it is often logged in a time series database. ![](https://i.imgur.com/QUUpRV5.png =600x) Some widely used tools are: * **Telegraf** is a metrics collection agent. Use it to collect and send metrics to InfluxDB. Telegraf’s plugin architecture supports collection of metrics from 100+ popular services right out of the box. * **InfluxDB** is a high performance Time Series Database. It can store hundreds of thousands of points per second. The InfluxDB SQL-like query language included in **InfluxDB V1** was built specifically for time series. InfluxQL is a query language that is very similar to SQL and that allows any user to query its data and filter it. In **InfluxDB V2** and **V3**, they make use of their new language **Flux**, which you can read more about [here](https://docs.influxdata.com/flux/v0/). Read more about InfluxDB in their documentation: [InfluxDB Documentation](https://docs.influxdata.com/) * **Grafana** is an open-source platform for data visualization, monitoring and analysis. In Grafana, users can to create dashboards with panels, each representing specific metrics over a set time-frame. Grafana supports graph, table, heatmap and freetext panels. ![](https://i.imgur.com/aANRGpe.png) In this exercise we will use the containers platform [Docker](https://www.docker.com/). Docker can be easily installed in in a large variety of platforms like a [Raspberry Pi.](https://www.raspberrypi.org/blog/docker-comes-to-raspberry-pi/) [Install Docker](https://docs.docker.com/compose/install/) for your operating system. Please note, if you are using Linux you need to manually install docker-compose, on desktop systems like Docker Desktop for Mac and Windows, Docker Compose is included as part of those desktop installs. We will use the following images: * https://hub.docker.com/_/telegraf * https://hub.docker.com/_/influxdb * https://hub.docker.com/r/grafana/grafana The images will be pulled automatically when running the docker-compose script, but they can also be manually pulled (and updated) with the command below: ```bash $ docker pull telegraf $ docker pull influxdb $ docker pull grafana/grafana ``` ## Setting up Mosquitto There are several ways to set up an MQTT broker. In this guide, we will focus on a local install of Mosquitto on a Raspberry Pi running Raspberry Pi OS. For **Windows** or **Mac**, see the Eclipse Mosquitto Download page [here](https://mosquitto.org/download/). There are installation guides for respective operating system in the README. An alternative to Mosquitto is **HiveMQ**. Read more about HiveMQ and how to install it [here](https://www.hivemq.com/). ### Update System Packages and Installing Mosquitto **Step 1:** Start by updating your system packages: ```=bash $ sudo apt-get update -y $ sudo apt-get upgrade -y ``` **Step 2:** Install Mosquitto ```=bash $ sudo apt-get install mosquitto mosquitto-clients -y ``` **Step 3:** Set up user authentication To make your broker secure, you need to set it up with a username and password. First, create a password file: ```=bash $ sudo mosquitto_passwd -c /etc/mosquitto/passwd username # Replace username with a name of your choice ``` **Step 4:** Edit the config file to allow your credentials: ```=bash $ sudo nano /etc/mosquitto/mosquitto.conf ``` Add the following lines to the config file: ```=bash allow_anonymous false password_file /etc/mosquitto/passwd ``` Then restart the Mosquitto service: ```=bash $ sudo systemctl restart mosquitto ``` **Step 5:** Test your Mosquitto broker ```=bash $ mosquitto_sub -h localhost -t test -u "username" -P "password" # Replace with your username and password $ mosquitto_pub -h localhost -t test -m "hello world" -u "username" -P "password" # Replace with your username and password ``` ### Docker-compose Compose is a tool for defining and running multi-container Docker applications, which in this case is three different containers. Instead of firing up all container individually with the respective config, we can write a script which does this for us. A YAML file is used to configure your application’s services. Then, with a single command, you create and start all the services from your configuration. Start all services with `docker-compose up`, and take them down with `docker-compose down`. Very easy. **Step 1:** Create a file `docker-compose.yml` in a working folder. **Step 2:** Use this template and add it to your `docker-compose.yml` file. Make sure to read through the template **thoroughly** and get an understanding of what needs to be altered later on before proceeding. ```=bash #version: '3.9' networks: tig-net: driver: bridge volumes: tig-data: services: influxdb: image: influxdb:latest container_name: influxdb ports: - 8086:8086 environment: INFLUXDB_HTTP_AUTH_ENABLED: "true" INFLUXDB_DB: "" # THIS WILL BE REPLACED INFLUXDB_ADMIN_USER: "" # THIS WILL BE REPLACED INFLUXDB_ADMIN_PASSWORD: "" # THIS WILL BE REPLACED networks: - tig-net volumes: - tig-data:/var/lib/influxdb grafana: image: grafana/grafana:latest container_name: grafana ports: - 3000:3000 environment: GF_SECURITY_ADMIN_USER: {GRAFANA_USER_NAME} # Replace this before starting the containers GF_SECURITY_ADMIN_PASSWORD: {GRAFANA_PASSWORD} # Replace this before starting the containers networks: - tig-net volumes: - tig-data:/var/lib/grafana restart: always telegraf: image: telegraf:latest depends_on: - "influxdb" environment: HOST_NAME: "telegraf" INFLUXDB_HOST: "influxdb" INFLUXDB_PORT: "8086" DATABASE: "" # THIS WILL BE REPLACED volumes: - ./telegraf.conf:/etc/telegraf/telegraf.conf tty: true networks: - tig-net privileged: true ``` Take note of this part, which you will have to alter **before** you start the containers. You can choose whatever username and password you want. These credentials are used to log in to Grafana. Which you can do on **127.0.1:3000**. ```=bash environment: GF_SECURITY_ADMIN_USER: {GRAFANA_USER_NAME} # Replace this before starting the containers GF_SECURITY_ADMIN_PASSWORD: {GRAFANA_PASSWORD} # Replace this before starting the containers ``` **Step 3:** Go to **127.0.1:8086** in your web browser, and follow the setup guide for InfluxDB: ![image](https://hackmd.io/_uploads/rkU9UP6S0.png) Be sure to save your **API token** for later. ![influx-token](https://hackmd.io/_uploads/B131f5TSR.png) Once having done this, stop the containers with `sudo docker compose down` and add your credentials to your `docker-compose.yml` file before starting the containers with `sudo docker compose up` again Replace the `INFLUX_DB` with the name of the **Bucket** you created in the InfluxDB setup process, and replace `{USERNAME}` and `{PASSWORD}` with the user name and password you chose in the setup process. ```=bash environment: INFLUXDB_HTTP_AUTH_ENABLED: "true" INFLUXDB_DB: "{BUCKET_NAME}" # THIS WILL BE REPLACED INFLUXDB_ADMIN_USER: "{USERNAME}" # THIS WILL BE REPLACED INFLUXDB_ADMIN_PASSWORD: "{PASSWORD}" # THIS WILL BE REPLACED ``` ## Configuring Telegraf You will need to configure Telegraf. Study the template below and put it in a filed called `telegraf.conf` in the same working directory as your `docker-compose.yml` file. Replace the corresponding variables to fit your personal setup. ```=bash [agent] flush_interval = "30s" interval = "30s" [[inputs.mqtt_consumer]] name_override = "{}" # Choose a name servers = ["tcp://{IP}:1883"] # REPLACE WITH IP OF YOUR BROKER qos = 0 connection_timeout = "30s" topics = [ "{TOPIC1}", "{TOPIC2}", "TOPIC.." ] # REPLACE WITH YOUR OWN TOPICS username = "{USERNAME}" # REPLACE THIS WITH MOSQUITTO USERNAME password = "{PASSWORD}" # REPLACE THIS WITH MOSQUITTO PASSWORD data_format = "value" # MIGHT NEED REPLACING data_type = "float" # MIGHT NEED REPLACING [[outputs.influxdb_v2]] bucket = "{BUCKETNAME}" # REPLACE WITH BUCKET NAME urls = [ "http://influxdb:8086" ] token = "{API-TOKEN}" # REPLACE WITH InfluxDB API TOKEN organization = "{ORGANIZATION}" # REPLACE WITH ORGANIZATION NAME ``` ## Setting up Grafana Go to **127.0.1:3000** in your browser, which will take you to Grafana. Log in with the credentials you set in the `docker-compose.yml` file. **Step 1:** Install the InfluxDB plugin: ![grafana-plugin](https://hackmd.io/_uploads/ryNFYa6HA.png) **Step 2:** Add a data source: ![image](https://hackmd.io/_uploads/BkBV96aHR.png) **Step 3:** Configure your data source: The values to be entered should be self explanatory, and will be the same as the ones you used in the previous steps. **Note:** You can choose between **InfluxQL** and **Flux** according to your liking. ![image](https://hackmd.io/_uploads/HyNO5pTBR.png) **Step 4:** Create a new dashboard: ![image](https://hackmd.io/_uploads/HJ4gip6SA.png) **Step 5:** Add a new panel: Take note of the **query**, in this example **Flux** is used. These queries must be altered to fit your individual setup. ![image](https://hackmd.io/_uploads/BkBPipprC.png) **Step 6:** You should now be ready for creating your own personal dashboard! <style> html, body, .ui-content { background-color: #333; color: #ddd; } body > .ui-infobar { display: none; } .ui-view-area > .ui-infobar { display: block; } .markdown-body h1, .markdown-body h2, .markdown-body h3, .markdown-body h4, .markdown-body h5, .markdown-body h6 { color: #ddd; } .markdown-body h1, .markdown-body h2 { border-bottom-color: #ffffff69; } .markdown-body h1 .octicon-link, .markdown-body h2 .octicon-link, .markdown-body h3 .octicon-link, .markdown-body h4 .octicon-link, .markdown-body h5 .octicon-link, .markdown-body h6 .octicon-link { color: #fff; } .markdown-body img { background-color: transparent; } .ui-toc-dropdown .nav>.active:focus>a, .ui-toc-dropdown .nav>.active:hover>a, .ui-toc-dropdown .nav>.active>a { color: white; border-left: 2px solid white; } .expand-toggle:hover, .expand-toggle:focus, .back-to-top:hover, .back-to-top:focus, .go-to-bottom:hover, .go-to-bottom:focus { color: white; } .ui-toc-dropdown { background-color: #333; } .ui-toc-label.btn { background-color: #191919; color: white; } .ui-toc-dropdown .nav>li>a:focus, .ui-toc-dropdown .nav>li>a:hover { color: white; border-left: 1px solid white; } .markdown-body blockquote { color: #fff; } .markdown-body table tr { background-color: #5f5f5f; } .markdown-body table tr:nth-child(2n) { background-color: #4f4f4f; } .markdown-body code, .markdown-body tt { color: #f; background-color: rgba(230, 230, 230, 0.36); } a, .open-files-container li.selected a { color: #5EB7E0; } .markdown-body pre { color: black; } </style>