IoT tutorial on monitoring weather data in your home using the TIG stack

Author: Alex Karlsson (ak223ke)

This IoT project is about monitoring and visualizing temperature, humidity, and light level data. To this end we use the TIG stack which consists of Telegraf, InfluxDB and Grafana. To measure the data, we use a DHT11 sensor and a KY-018 light sensor connected to a LoPy4 device. The data is sent from the device to a MQTT broker over a WiFi connection and then inserted into the database InfluxDB by Telegraf. Grafana is then used to visualize the data and to set alarms which notifies us when the temperature rises above a certain degree. A WiFi connection is used since the main objective is to measure data indoors.

This project should take around 5-9 hours to complete depending on experience. This project is not that advanced, but it consists of multiple parts such as connecting the device, programming the device and Docker. As such there is much to learn which is what I think makes it such a fun project!

Objective

There are two main reasons that I chose to do this project and both reasons are primarily related to temperature data. Reason 1) is that I suspect that my apartment is colder during the winter and hotter during the summer than the recommendation by the Public Health Authority in Sweden [1]. Reason 2) is that I want to make sure that the temperature inside is acceptable for my dog. I'm also interested to see how my dog's energy level corresponds with the temperature.

The reason for monitoring humidity is simple, I'm interested in having a good climate indoors [2]. For light levels, I'm only interested in seeing how it correlates to the temperature.

The purpose with this IoT project is to monitor and visualize temperature, humidity and light levels for the reasons stated above. Another purpose is setting alarms which we can do with Grafana. Alarms can alert us with a notification when the temperature reaches a certain degree.

This project can provide insights into primarily temperature and humidity data over time. The visualized measurements can show if the temperature is at recommended levels and indicate if we have a good climate indoors.

Material

The materials used in this IoT project can be seen in Table 1. All the material was purchased in a bundle from the web shop Electrokit. Note that all princes link to a web shop in which the product can be purchased. Also, that prices in SEK have been converted to EUR on the 29-07-2021 and that there can be minor differences to these prices depending on currency fluctuation.

Item	Price	Purpose
Pycom LoPy4 Device	€38.45	This is the IoT device. It is programmable with MicroPython and supports WiFi, LoRa, SigFox and Bluetooth connectivity.
Pycom Expansion Board 3.1	€17.60	General purpose expansion board that adds functionality to the LoPy4 device. Most notably a USB cable is connected to the expansion board which powers the device, and the wiring is connected to the pins on the expansion board.
Breadboard	€5.80	Makes it easier to connect the device with the sensors. Not essential but improves quality of life.
DHT11 sensor	€4.82	Measuring temperature and humidity.
KY-018 Light sensor	€3,84	Measuring light level.
	Total
	€70.51

Table 1: Materials

Cables are not included in Table 1, but these should be cheap. Furthermore, the prices don't include shipping. As a final note the listed price of the expansion board is for v.3.0 since I was unable to find v.3.1 on Pycom's website. However, the price should not differ much between these versions.

Data sheets and additional information about the Pycom products and the sensors can be found with simple searches online.

Images of the material can be seen in Figure 3 (Fritzing diagram), 7, 8 and 9. If you want a closer look at something you can click on the links in the price column. You can also find images of all the material online.

Computer setup

This IoT project has been done on a Windows 10 PC. However, based on what I have learned I think that the setup should be similar on Mac and Linux operating systems.

Software

In this IoT project I have used the software seen in Table 2. The software is linked in the table and all of the software is free and can be found online with ease. The software and Pymakr plugin must be installed on your machine to complete the tutorial. The steps for installing the Pymakr plugin for VS Code will be detailed later in this tutorial. You should not experience problems installing the other software since the executable files walks you trough the installations.

Note that the Pymakr plugin can also be installed on Atom which is another popular code editor. In the end it's a matter of preference and I chose to use VS Code since I have prior experience with it.

Software	Purpose
Visual Studio Code (VS Code)	Code editor used for programming in Python and for editing Docker related files
Docker	Docker Compose is used to run the TIG stack
Pymakr (VS Code plugin)	Plugin by Pycom to upload, download and run code on the LoPy4 device
Pycom Firmware Updater	Program by Pycom to update and flash the device

Table 2: Software used in the project

Note: Python and Node.js [3] is required for the Pymakr plugin. The latest version of Python can be installed here and Node.js here. Both need to be added as PATH environment variables so that shells, e.g., Git Bash and PowerShell, can access them. On Windows 10, adding them to PATH is an option during installation. However, this can be done manually as well by editing the system environment variables.

Installing extensions in VS Code

The following steps can be followed to install the Pymakr plugin (or rather extension as seen in Figure 1) in VS Code.

Start VS Code
Navigate to the menu option called extensions
Search for the extension (Pymakr)
Install the extension
Restart VS Code

Figure 1: Installing Pymakr

Programming and uploading code to the device

MicroPython is used when programming for the LoPy4 device. It is described as a leaner and more efficient implementation of Python 3 [4]. MicroPython's syntax is identical to Python which is a high-level language that I think is pleasant to work with. However, I think that it's essential to know that it does not include all of Python's standard modules [5] and I have found it doesn't work with the pip package manager. The included Pycom and MicroPython modules can be found in the Pycom documentation.

The Pymakr extension is used to run, upload, and download code from the device. In VS Code, these options are presented in a bar at the bottom of the editor. Above this is the Pymakr Console (terminal) which can be used to access the REPL in order to execute code directly on the device. These features make working with the device both fast and efficient. Both the bar containing the options and the terminal to access the REPL can be seen in Figure 2 below.

Figure 2: Pymakr extension

Updating and flashing the device

The tool Pycom Firmware Updater can be used to update and flash the device. This can be useful for multiple reasons, such as to change or update the firmware or to erase all files uploaded to the device. The Pycom documentation has a download link for the tool and all the information you'll need.

Putting everything together

The wiring can be seen in Figure 3. The LoPy4 pinout diagram and data sheets of the sensors (can be found online) are essential when connecting the device with sensors. The pinout shows what each pin support, e.g., ADC (analog to digital), and the data sheet of the sensors show operating voltage and what each pins on the sensors are for, e.g., ground and signal. How to connect the sensor pins can also be derived from looking at the sensor, in Figure 3 we can see that the sensors have S for signal and – for ground. Worth mentioning is that connecting something wrong could damage the sensor and even the device if you're unlucky.

The sensor to the left on the breadboard is the DHT11 sensor. This sensor is connected to P23 on the expansion board. To the right we have the light sensor which is connected to P20 on the board. Each sensor also has 3.3V operating voltage and ground supplied from the device. As seen on online resources both sensors supports 3.3V operating voltage [6]-[7].

For the purposes stated before, I think that this can be used in production. With that said, there are improvements to be made related to both hardware and software. I'm sure software improvements to be made will be discovered over time. For hardware, the DHT11 sensor is limited in that it only returns whole numbers and not decimals, e.g., 26.3 °C. For this reason, I would like to replace it with another temperature sensor in the future that is more precise.

Figure 3: Wiring

Platform

TIG Stack

The platform I have chosen for this project is the TIG stack (Telegraf, InfluxDB and Grafana). The purpose of each letter in the stack is explained in Table 3 below:

TIG Stack	Purpose
T	Telegraf is used to fetch messages from the MQTT broker and insert them into the database
I	InfluxDB is a time series database that is used to store the data
G	Grafana is used to visualize the data and also supports features such as alarms and annotations

Table 3: TIG Stack

As described, Telegraf is used to fetch messages from the MQTT broker. These messages are sent to the broker by the MicroPython code that runs on the device. Figure 4 is an informal diagram that I created which shows how the parts in the infrastructure are connected.

Figure 4: TIG Stack

Docker Compose is used to run the TIG stack on our local machine. Describing Docker and Docker Compose in-depth is out of scope for this tutorial. However, in short, with Docker we don't have to download and install various software. Instead, Docker downloads the software in the form of Docker images and runs it as processes in containers on our machine [8]. The infrastructure can be seen in the Docker Compose file on GitHub and can be started and stopped using the two commands listed below:

docker-compose up
docker-compse down

I have chosen the TIG stack since it's a local solution that's free and doesn't depend on external parties, e.g., a cloud provider. The solution supports our use case and Docker makes it simple to run the services. However, there's nothing preventing one from using the TIG stack hosted at a cloud provider.

However, it should be noted that in this case I depend on an external MQTT broker (supplied by the course administration), but it's possible to setup your own local broker on IoT devices such as a Raspberry PI.

To conclude, I have also experimented with Grafana provisioning. You can read more about provisioning here, but, in short, with provisioning can data sources, dashboards, panels and more be created and made available as soon as the service is started. While I also make use of volumes to make data persistent someone that have not started the services before would be greeted with a working data source and a dashboard.

The code

The code can be found on GitHub. It should be noted that some of the code is from course resources and examples that are also found on the course's GitHub. I have made sure to comment the source in all the code related files that are based on external resources. This is the case for the files in the lib folder (DHT11 and MQTT related), boot file and to some extent the Docker Compose file.

As stated, the code can be found on GitHub. However, I also describe some of the important code below.

The code block below is from the boot file which is the first file to run on boot up. This file connects to WiFi using credentials loaded from a config file (not to be included in version control). Note that the config file also contains other credentials such as for connecting to the MQTT broker (not seen in the code block below). As stated before, the boot file is from a course example.














wlan = WLAN(mode=WLAN.STA)

with open('config.json') as f:
    config = json.load(f)

nets = wlan.scan()
for net in nets:
    if net.ssid == config['ssid']:
        print('Network found!')
        wlan.connect(net.ssid, auth=(net.sec, config['ssid_pass']), timeout=5000)
        while not wlan.isconnected():
            machine.idle() # save power while waiting
        print('WLAN connection succeeded!')
        break

The code block below is from the main file and shows us the function which publish the data to the MQTT broker. Data is added to a what is known as a Dictionary in Python and then converted to JSON before published. An exception is raised if the data from the DHT11 sensor is not valid.













def send_data():
  dict = {'home_sensor_1': {}}
  dht_result = dht_sensor.read()
  light_result = light_sensor.read()
  if (dht_result.is_valid()):
      dict['home_sensor_1'].update({ 'temperature': dht_result.temperature })
      dict['home_sensor_1'].update({ 'humidity': dht_result.humidity })
  else:
    raise Exception('DHT Error: ', dht_result.error_code)
  # No error checking implemented for the light sensor
  dict['home_sensor_1'].update({ 'light': light_result})
  # Publish the dictionary over MQTT as JSON
  client.publish(TOPIC_PUB, json.dumps(dict))

The code block below is from the main file and shows us what I call the main loop and exception loop. The main loop is executed forever unless there is an unhandled exception. If there's an unhandled exception, the exception loop makes a maximum of 3600 attempts (which should take one hour) to send the data successfully. If one attempt is successful, we are returned to the main loop. However, if all attempts fail the program terminates.






















# Exception loop, exit on 3600 attempts (1 hour)
def exception_loop():
  for i in range(3600):
    try: 
      send_data()
      return # Return on success
    except Exception as e:
      print('Error: ', e)
    finally:
      time.sleep(1)
  # Exit on failed retries
  sys.exit()

# Main loop
while True:
  try:
    send_data()
  except Exception as e:
    print('Error: ', e)
    exception_loop()
  finally: 
    time.sleep(SLEEP_INTERVAL_SECONDS)

Transmitting the data / connectivity

The data is transmitted every 600 seconds (ten minutes). However, I think it would be fine to receive the data every 30 minutes as well. How often data is sent can be modified by updating one variable in the code.

The data is sent over WiFi. The main reason for using WiFi and not e.g., LoRa, is that the main purpose is to measure data indoors. Since 98% of the Swedish population have internet access at home connectivity is almost guaranteed [12].

The data is sent using the transport protocol MQTT. This is suitable for IoT since it is lightweight, scalable, has features to ensure reliable message delivery and bi-directional communication [13]. As an example, these features can be important for handling situations in which messages don't arrive.

The data is published on the MQTT broker under the topic: ak223ke/devices/weather/

The data is sent as JSON, and the structure can be seen below:







{
  "home_sensor_1": {
    "humidity": 44,
    "temperature": 26,
    "light": 178
  }
}

Presenting the data

The data is saved to the database as soon as Telegraf has detected a new message published under the topic at the MQTT broker. As stated before, since data is sent every ten minutes, new data is saved around every ten minutes as well.

As mentioned before, the database used is called InfluxDB. This is a suitable choice since it's a time series database [14] optimized for time-stamped [15] data. Operations on the time such as sorting is often done on measurements such as the data sent from our device. Another example of a time series database is Timescale.

Grafana is used to visualize the data. Seen in Figure 5 is a dashboard containing various panels that displays the data. The panel at the top is called a gauge and shows the most recent value received. Colours are used to visualize the values, e.g., yellow between 25 and 30 °C. Below the gauges we have normal line charts.

Figure 5: Grafana dashboard

Grafana also has an alarm feature that I make use of to send out notifications. As seen in Figure 6 I have configured it to send out a Discord notification when the temperature reaches 27 °C.

Figure 6: Grafana alarm (Discord notification)

Finalizing the design

Overall, I'm pleased with the results of this project. The code running on the device is working as expected and data is being reported with the help of the TIG stack. I'm also a fan of Grafana which is a powerful tool for visualizing the received data.

However, as mentioned before, I would like a sensor that can report more precise temperature with decimal values. So, testing another temperature and thus humidity sensor as well (since DHT11 is both) is something to be done in the future.

Also, I would like to setup a local MQTT broker in the future in order to experiment more with reliable message delivery and bi-directional communication.

To improve the visualization of the data I would also like to create a panel that combines temperature and humidity data. This is something I had not considered and while minor I think it would add much value to the visualization part.

Figure 7, 8 and 9 shows the final design from different angles.

Figure 7: Final results 1

Figure 8: Final results 2

Figure 9: Final results 3

Referencing

When it comes to referencing, I have combined IEEE with normal links. The reason for this combination is that I don't think scientific referencing is suitable for a practical tutorial in this case. As such I have also used normal links to resources found on the web.

I have tried to separate the two with the following reasoning. Proper referencing with IEEE has been reserved to claims and statements. Links to web resources have been used so that the reader of the tutorial can learn more about a particular topic or service. The distinction could be considered unclear in some borderline cases, but I hope the combination is ok and found suitable for this tutorial.

References

[1] ‘Temperatur inomhus — Folkhälsomyndigheten’. http://www.folkhalsomyndigheten.se/livsvillkor-levnadsvanor/miljohalsa-och-halsoskydd/tillsynsvagledning-halsoskydd/temperatur/ (accessed Jul. 27, 2021).

[2] ‘Inomhusklimat - temperatur, koldioxid och luftfuktighet inomhus - Intab’. https://intab.se/radgivning/inomhusklimat (accessed Jul. 27, 2021).

[3] ‘Visual Studio Code’. https://docs.pycom.io/gettingstarted/software/vscode/ (accessed Jul. 27, 2021).

[4] ‘MicroPython - Python for microcontrollers’. http://micropython.org/ (accessed Jul. 28, 2021).

[5] ‘1. What Is MicroPython? - Programming with MicroPython [Book]’. https://www.oreilly.com/library/view/programming-with-micropython/9781491972724/ch01.html (accessed Jul. 28, 2021).

[6] ‘Temperature and Humidity Sensor Module’. https://www.electrokit.com/uploads/productfile/41015/41015728_-_Digital_Temperature_Humidity_Sensor.pdf (accessed Jul. 29, 2021).

[7] ‘KY-018 Photoresistor Module’, ArduinoModulesInfo, Feb. 16, 2017. https://arduinomodules.info/ky-018-photoresistor-module/ (accessed Jul. 29, 2021).

[8] ‘Docker overview’, Docker Documentation, Jul. 30, 2021. https://docs.docker.com/get-started/overview/ (accessed Jul. 29, 2021).

[9] ‘AWS IoT - Amazon Web Services’, Amazon Web Services, Inc. https://aws.amazon.com/iot/ (accessed Jul. 29, 2021).

[10] ‘Azure IoT Central | Microsoft Azure’. https://azure.microsoft.com/en-us/services/iot-central/ (accessed Jul. 29, 2021).

[11] ‘Google Cloud IoT - Fully Managed IoT Services’, Google Cloud. https://cloud.google.com/solutions/iot?hl=sv (accessed Jul. 29, 2021).

[12] ‘98 procent har tillgång till internet hemma’, Svenskarna och internet. https://svenskarnaochinternet.se/rapporter/svenskarna-och-internet-2018/allmant-om-internetutvecklingen/98-procent-har-tillgang-till-internet-hemma/ (accessed Jul. 30, 2021).

[13] ‘MQTT - The Standard for IoT Messaging’. https://mqtt.org/ (accessed Jul. 30, 2021).

[14] ‘InfluxDB: Purpose-Built Open Source Time Series Database’, InfluxData. https://www.influxdata.com/ (accessed Jul. 30, 2021).

[15] ‘Time Series Database (TSDB) Explained | InfluxDB’, InfluxData. https://www.influxdata.com/time-series-database/ (accessed Jul. 30, 2021).