How to build a in-door weather and soil monitoring device using Micropython and the TIG-stack

Author: Laoa Jalal (lj223ep)

This project is about monitoring the humidity, temperature and soil moisture of in-door plants with the help of various sensors and a microcontroller. The data is sent through a WiFi connection to an external broker which then forwards the data to a database. The data is then converted to a graphical visualization that represents the temperature, humidity and soil moisture in a time series.
Comment
The tutorial will show the necessary components needed to replicate the following procedure, as well as an explanation of the various techniques used to achieve the end goal.
The project is estimated to take 4-6 hours depending on the reader's experience. The project itself is straightforward, but because of the use of many technologies, there are also many pitfalls. However, the use of docker simplifies things immensely as we will see later on.

Objective

The main purpose for choosing this project is that my mother loves her plants, (almost more than me I think), and because she has a lot of them, it can become tedious to know when a plant needs watering, especially in the summer where the heat can rise to very high levels.

The project will help her keep track of the soil moisture of her favourite plants and know exactly when to water her plants at the right time, and when the plants have received just enough water. To achieve this, she will be notified by email if the soil moisture in the plant is too low/high.

The project is mainly focused on monitoring in-door plants but can be easily extended to handle outdoor monitoring. The reason for this limitation is that no casing has been created for the module which protects it from external weather phenomenons such as rain and wind.

Material

The materials used in this project:

Hardware	Price	Description	Link
WIFI LoRa 32 Heltec	35,88$	IoT device that supports WiFi and LoRa connection. Can be programmed with micropython and has many GPIO pins.	Heltec
LNU – 1DT305 Tillämpad IoT – Sensors only bundle	12.48$	A sensor bundle containing a breadboard, resistors, jumper wires and some sensors, which are not used in this particular project.	Sensor only bundle
Soil moisture sensor	6,77$	A sensor that is used to measure the moisture level of soil.	Soil moisture sensor
DHT11 temperature and humidity sensor module	4.74$	A sensor that measures both temperature and humidity	DHT11
Soldering iron (optional)	30$	For soldering header pins on heltec device	Soldering iron

The total cost for the equipment: 89.87$ or 59.87$ without the soldering iron.

Computer Setup

The setup is a mixture of using Windows 10 as the host machine alongside Ubuntu 18.0.4 LTS running on a virtual machine. The virtual machine has the responsibility of setting up the TIG-stack as well as setting up the broker software. The windows machine is used for flashing and programming the device (Heltec).

Windows 10

Flashing Heltec Firmware

When flashing the Heltec firmware, you have to first install the hardware driver for a generic esp 32 board. You can download it here. The zip file contains files that ends with _x64 and _x86. If you have a modern computer, press the one ending with _x64.

After executing the file, you should see a new Port appear in the Device Manager named "Silicon Labs CP210x" when plugging the USB from the computer to the heltec device.

Now, download this and visit this website.

The website will look like this:

• First, select 460800 Baud, then press connect
• After pressing connect, a pop-up window should appear. Choose the port that was previously mentioned (CP2102 ..) and press connect.
• Now, a new panel will appear looking like this
  
• Press the erase button and let wait for the device to be fully erased.
• Enter 1000 inside one of the offset input field and press choose a file for that offset field. Enter the previously downloaded file named esp32-20220117-v1.18.bin. The file may have a different name on your system.
• Press the program button. Wait for the device to be fully programmed and do not remove the USB cable
• You are now done!

You can find the original post which i have reffered to here which also covers how you flash the esp32 on other operating systems.

Chosen IDE

The IDE used for programming the heltec device is atom which you can download here.

Software

After downloading atom, we need to install Node.js for the pymakr plugin to work. Download the latest version of Node.js here.

Now open atom and either navigate to the package installer by either pressing File > Settings > Install or often times, when you have freshly installed atom, a tab named "Welcome Guide" will appear which provides a direct navigation to the package installer. Inside the package installer, search for pymakr and press install.

After installing the plug-in, navigate to File > Settings and press on the packages. Hopefully, you will find the newly installed package, looking like this:

Press on Settings and navigate to the field which asks for the Device addresses. Now, if you have a good memory you know the port number, if not its ok, just go to the device manager and look for the port number, mine is COM3 just as depicted in the figure below. Just a quick reminder, you have to connect your heltec device to your computer to see the port appear inside the device manager.

Write the port number inside the Device addresses field and close the settings tab.

On the bottom right corner of atom, a tab named pymakr is present, press on this tab. You should now see the following panel depicted below

Press on "Connect device" and find the port number that you previously inserted inside the device addresses. You should now be able to upload files and program your heltec device!

Putting everything together

The circuit diagram for the project is represented below

• The black cables represents connection to ground
• The red cables are high voltage cables, in this case 3.3V
• The yellow cables are the data cables. In this case, the DHT11 module is connected to PIN 13 on the heltec device and the soil sensor to PIN 32.
• The resistors used for the analog output pin are there for creating a voltage divider when the analog data that is sent by the soil moisture to the heltec device. I use this setup because the analog pin from the sensor module does not regulate the voltage output. If i for instance connect the sensors with 5V, a 5V output from the soil sensor could damage the heltec device.
• The resistor connecting to the analog wire (blue) has a resistance of 20k ohm while the resistor connected to the ground has 10k ohm. This will provide a voltage of 1.1V when a input of 3.3V is used.

Platform

The project uses the TIG-stack for storing and visualizing the data the sensors gather. Using the TIG-stack, in contrast to other cloud-based solutions, enables you to set up a local way of storing and handling the data without the need for third-party services. Third-party services often restrict the user from properly scaling their solution. To get the full features, you are often required to pay money.

A good guide for gettings started using the TIG-stack can be found here which uses docker for an easy deployment in most environments.

The message broker used in this project is the open-source eclipse mosquitto software. Mosquitto implements the MQTT protocol which enables low-power devices to subscribe and publish messages to the broker.

The intended structure of the project:

The environment used for the following procedure is Ubuntu 18.04.6 LTS

Mosquitto

Follow this tutorial on how to download and configure the mosquitto broker.
The tutorial also has a great walkthrough on how to set up a SSL certificate with the help of let´s encrypt to securely communicate to/from the broker.

After following the tutorial you should be able to have a running MQTT broker.

My mosquitto configure file looks like this:

persistence false
allow_anonymous true

password_file /etc/mosquitto/passwd

acl_file /etc/mosquitto/acl

# mqtt

#listener 1883
listener 8883
certfile /etc/letsencrypt/live/mqqtserverhome.duckdns.org/cert.pem
cafile /etc/letsencrypt/live/mqqtserverhome.duckdns.org/chain.pem
keyfile /etc/letsencrypt/live/mqqtserverhome.duckdns.org/privkey.pem
protocol mqtt

The MQTT broker listens at port 8883 (SSL). If you decide to not use SSL, use port 1883 instead.

Very important:

• If you use the broker as me, that is, as an external service that can be located anywhere, you have to allow traffic from port 1883/8883 on the system firewall. To do this, write sudo ufw allow 8883 or sudo ufw allow 1883 in the terminal.
• To allow the broker to be accessed outside the LAN, login to the router gateway and allow port-forwarding for the device that runs the MQTT service.

The above figure shows that my router has allowed port forwarding for 3 services located at the IP-address 192.168.1.7 (ignore the HTTPS_APACHE row).

TIG-stack

Before starting, you first have to install docker. When docker has been installed, write the following commands in the terminal:

$ docker pull telegraf
$ docker pull influxdb
$ docker pull grafana/grafana

The following commands downloads the docker images that is used in this project. Now, create a new directory and inside the directory, create 2 files named docker-compose.yml and telegraf.conf.

Insert the following to the docker-compose file:

version: '3.9'

networks:
  tig-net:
    driver: bridge

volumes:
  tig-data:


services:
  influxdb:
    image: influxdb:1.8.6
    container_name: influxdb
    ports:
      - 8086:8086
    environment:
      INFLUXDB_HTTP_AUTH_ENABLED: "true"
      INFLUXDB_DB: "iot"
      INFLUXDB_ADMIN_USER: "iotlnu"
      INFLUXDB_ADMIN_PASSWORD: "micropython"
    networks:
      - tig-net
    volumes:
      - tig-data:/var/lib/influxdb
     
  grafana:
    image: grafana/grafana:7.5.9
    container_name: grafana
    ports:
      - 3000:3000
    environment:
      GF_SECURITY_ADMIN_USER: king
      GF_SECURITY_ADMIN_PASSWORD: arthur
      GF_SMTP_ENABLED: "true"
      GF_SMTP_HOST: "smtp.gmail.com:465"
      GF_SMTP_USER: "" # Insert email address of notifier.
      GF_SMTP_PASSWORD: "" # password of email address that notifies users.
      GF_SMTP_FROM_ADDRESS: "" # Same as GF_SMTP_USER
      GF_SMTP_FROM_NAME: "Grafana"
      GF_SMTP_SKIP_VERIFY: "false"
      
    networks:
      - tig-net
    volumes:
      - tig-data:/var/lib/grafana
    restart: always

  telegraf:
    image: telegraf:1.19.0
    depends_on:
      - "influxdb"
    volumes:
      - ./telegraf.conf:/etc/telegraf/telegraf.conf
    tty: true

The docker-compose file contains information about:

• Which services to start, in this case, telegraf, influxdb and grafana.
• The ports used to access respective services.
• Enable SMTP for grafana, used for notifying user about critical state (will be discussed in later section)
• log-in credentials for both influxdb and grafana.
• specifing docker image volumes.

Inside the telegraf.conf file, insert the following:

[agent]
  flush_interval = "15s"
  interval = "15s"

[[inputs.mqtt_consumer]]
  name_override = "mqqt_mosquitto"
  servers = ["ssl://mqqtserverhome.duckdns.org:8883"] 
  qos = 0
  connection_timeout = "30s"
  topics = [ "weather/#" ]
  username = "laoa"
  password = ""
  data_format = "json"
  tls_ca = "/etc/ssl/certs/lets_encrypt/chain.pem"
  tls_cert = "/etc/ssl/certs/lets_encrypt/cert.pem"
  tls_key = "/etc/ssl/certs/lets_encrypt/privkey.pem"
  insecure_skip_verify = false

[[outputs.influxdb]]
  database = "iot"
  urls = [ "http://mqqtserverhome.duckdns.org:8086" ]
  username = "iotlnu"
  password = "micropython"

NOTE: use your own DNS-name/IP-address

• Inside the server field, insert either your IP-address/DNS-name and the port where the broker is located at. If you don't have ssl enabled, use tcp://.. instead of ssl://..
• If you don't use ssl, remove the tls_ca, tls_cert and tls_key fields.
• The topics field has to be the same as specified in the mosquitto acl configuration file.

Start services

Mosquitto

To start mosquitto, type sudo mosquitto -c /etc/mosquitto/conf_d/my_conf.cfg to the terminal

TIG-stack

To start telegraf, influxdb and grafana, locate to the directory which contains the docker-compose.yml file and type sudo docker-compose up in the terminal.

Code

The following code can also be found at github.

Boot.py

import config
import time
import ujson
import machine
from lib.mqtt import MQTTClient

def do_connect():
    import network
    sta_if = network.WLAN(network.STA_IF)       # Put modem on Station mode
    if not sta_if.isconnected():                # Check if already connected
        print('connecting to network...')
        sta_if.active(True)                     # Activate network interface
        sta_if.connect(config.SSID, config.WPA2_PASSWORD)     # Your WiFi Credential
        # Check if it is connected otherwise wait
        while not sta_if.isconnected():
            pass
    # Print the IP assigned by router
    print('network config:', sta_if.ifconfig())

def sub_cb(topic, msg):
    print("Received msg")

def restart_and_reconnect():
  print('Failed to connect to MQTT broker. Reconnecting...')
  time.sleep(10)
  machine.reset()

def listen_mqqt_server(client):
    while True:
        try:
            client.check_msg()
        except OSError as e:
            restart_and_reconnect()
        time.sleep(1)

def start_mqtt_connection():
    try:
        client = connect_mqqt_server()
        #listen_mqqt_server(client)
        return client
    except OSError as e:
        restart_and_reconnect()

def connect_mqqt_server():
    client = MQTTClient(config.MQTT_CLIENT_ID,
                    server=config.MQTT_BROKER,
                    user=config.MQTT_USER,
                    password=config.MQTT_PASSWORD,
                    port=config.PORT,
                    ssl=True)
    client.set_callback(sub_cb)
    client.connect()
    client.subscribe("weather/temp")
    print('connected to MQTT broker')
    return client



do_connect()
from main import read_sensors_and_publish
broker = start_mqtt_connection()
read_sensors_and_publish(broker)

main.py

from machine import Pin, ADC
from time import sleep
import dht
import ujson


'''
Calibrated soil estimate:

3.3V input

usage of volt dividers of 20k(ohm) and 10 k(ohm) which generates
an output voltage of 1.1V

Max ~ 850mV (dry)

Low ~ 250mV (in water)

'''

MAX_VOLTAGE = 850
MIN_VOLTAGE = 250

def read_sensors_and_publish(mqqt_broker):
    sensor_dht11 = dht.DHT11(Pin(13, mode=Pin.OPEN_DRAIN))
    pin = Pin(32, mode=Pin.IN)
    adc = ADC(pin, atten=ADC.ATTN_2_5DB)
    adc.width(ADC.WIDTH_10BIT)
    soil_sensor = adc

    while True:
        try:
            sleep(2)
            sensor_dht11.measure()
            temp = sensor_dht11.temperature()
            hum = sensor_dht11.humidity()

            moisture_in_soil = (MAX_VOLTAGE - soil_sensor.read()) * 100 / (MAX_VOLTAGE - MIN_VOLTAGE)
            print('Soil value: '+ str(moisture_in_soil) + "%")
            print('Temperature: %3.1f C' %temp)
            print('Humidity: %3.1f %%' %hum)

            publish_sensor_data(mqqt_broker, create_json(temperature=temp, humidity=hum, soil_moisture=moisture_in_soil), "weather")
        except OSError as e:
            print(e)
            print('Failed to read sensor.')


def create_json(**kwargs):
    try:
        print(dict(kwargs))
        return ujson.dumps(dict(kwargs))
    except Exception as e:
        return None

def publish_sensor_data(broker, data, topic):
    try:
        print(data)
        broker.publish(topic=topic, msg=data)
    except Exception as e:
        print("failed sending sensor data..")

conf.py

SSID = '' # Router name
WPA2_PASSWORD = '' # router password


MQTT_BROKER = '' #IP/DNS-name of MQTT broker

# If broker has login required, insert name and password
MQTT_USER = '' 
MQTT_PASSWORD = ''
PORT = '8883' # For ssl, if no ssl, use 1883 instead (or whatever port the MQTT is running at)

MQTT_CLIENT_ID = "id-1"

boot.py

• Connects the device to WiFi
• Connects to MQTT broker
• calls main.py function read_sensors_and_publish(broker) where the broker is injected to the function parameter.

main.py

• access the data-pins that transfer sensor data
• Loops indefinetly in which:
  1. The sensor data from dht11 is attained.
  2. The the soil moisture value is calculated (between 0-100)
  3. The data is packed to ujson format
  4. The ujson object is sent with subject "weather" to the broker.
  5. sleeps for 1 minute.
  6. repeat

used libraries:

dht.py

• A library from micropython that provides an interface to easily attain the sensor values for the dht11 in proper format.

mqtt.py

• Written by pycom
• Provides an interface for connecting and managing the connection to the message broker.
• methods for publishing and subscribing to the message broker.

Transmitting the data / connectivity

The device uses WiFi for transmitting data. The motivation behind using WiFi instead of other wireless protocols such as LoRa is that no LoRa gateway was present at the time.
The data is sent in JSON format with the MQTT protocol every minute using SSL. The use of SSL prevents eavesdropping on data (data becomes encrypted).

The use of WiFi instead of LoRa can be an issue when considering the battery life if one wishes to connect the device with a battery. However, this project is intended for in-door monitoring of plants so oftentimes, a power source outlet is almost always present at ease.

Presenting the data

When presenting the data, grafana is alongside influxdb. Grafana gives us a wide range of options on how we want to present the data. To present data, we need data. That is why we connect grafana with influxdb so the data can be obtained.
There is no limit set on how much data grafana can store, and the data is saved to the database each time the device publishes to the MQTT broker.

influxdb

I decided to use Influxdb because:

1. It is fundamentally included in the TIG-stack
2. It is open source and free to use
3. SQL-like syntax. This makes it easy because I carry previous experience with SQL
4. Because i can set-up my own database, I manage all security aspects and do not have to rely on a third-party solution.

To connect influxdb with grafana, look at this guide and navigate to the section Visualizing data with grafana.

Panels

The panels created is displayed below. A total of 5 panels was created. One for temperature, humidity and the last 3 for the soil moisture. The reason I created 3 panels for the soil moisture is because:

• We can only create 1 trigger per panel. I have decided to create 2 triggers for when the soil moisture is either too low/high.
• I want a panel which displays the current measured soil moisture (the gauge).

Triggers

I have decided to create a trigger for when the soil moisture is either too high or low. The low values is set to 10 or lower and a high value of 80 and higher. The notification is sent by email. That is why we enable SMTP inside the docker-compose.yml file.

The host used to send e-mail is gmail, click here to create a new gmail account.

Follow these steps to properly configure your gmail account to handle third-party access of the service:

• First, you have to insert the newly created gmail information to the docker-compose.yml file under the grafana section. Write the gmail address to the fields GF_SMTP_USER and GF_SMTP_FROM_ADDRESS.
• Go to your newly created gmail account and follow this guide.
• After creating the app password, write the password to the GF_SMTP_PASSWORD field.
• Save and re-run the TIG-stack.

Create trigger

To access grafana, write localhost:3000 to your web browser of choice. Log in to grafana using the credentials that was written inside the docker-compose.yml file (GF_SECURITY_ADMIN_USER and GF_SECURITY_ADMIN_PASSWORD field).

Head to the Alerting icon and press on notification channel. Press on new channel and name the channel to whatever suites your needs. Change the type to email and write the e-mail address of whatever address you wish to be notified by the trigger. Press save.

Add an alert to a panel, mine look like this:

When running device for 5 min with the soil moisture stick exposed to only air, the following mail is received from grafana:

This indicates that everything works as intended.

Finalizing the design

Images of end result:

Grafana panel when monitoring plant that was recently watered:

Conclusion

The project has been one of the more fun hands-on project i've done. I have learned a lot about IoT in whole but also gained much more intress in the area, which probably will result in me doing more fun IoT related projects in the future.

Improvements/Future work

• Use LoRa instead of WiFi. This will enable me to monitor plants further away from home.
• Create/buy a case for the device. This will enable protection against rain and other weather conditions which implies that we also can use it outdoors.
• Buy better soil moisture and temperature sensors. Ecspecially the soil moisture sensor as it is documented that the soil moisture stick corrodes after sometime which eventually makes it unfunctional.
• Buy a battery as an external power source. This further enables position independence (within the WiFi area of course).