---
title: Soil moisture monitoring and automated alerts
tags: plants, IoT
description: A monitoring system for your plants that keeps track of soil moisture content
---

# Soil moisture monitoring and automated alerts

#### Course: 1DT305 <br> Student: Essie Raza (er223fi)

This project aims to simplify the monitoring of plants and to provide a more reliable way of keeping them alive by sending automated watering alerts. These alerts will be triggered upon soil moisture levels reaching a certain threshold. It may also serve as a precursor to eventually upgrading to a fully automated watering system with minimal human interaction.

###### Estimated times to complete
No prior experience: 6h or more depending on issues that arise
Experienced: Under 6h

## Objective

This project was chosen to help me maintain healthier plants, as it will simplify the monitoring of them.

Watering plants on a regular schedule is harder than it sounds, especially when factors such as temperature and humidity already affect their moisture levels. It's easy to end up with a beloved plant that is either overwatered or neglected.

By eliminating all unknowns and instead watering them based on measured values provided by sensors, I will ensure that the plants receive just the right amount of moisture according to their individual needs.

It may also provide a good basis for a fully automated watering system in the future which draws from historical data of moisture measurements.

## Material / Resources

The materials were partly already in possession and partly purchased from resellers Electrokit.com and az-delivery.de. I use an ESP32 Heltec board with MicroPython code in this project to implement functionality and connectivity for measuring and sending moisture data from the sensors.

The moisture sensors I have chosen to work with are of the capacitive kind which are preferred over the more common two-pronged resistive ones. These sensors are much less prone to erosion due to having no exposed metal electrodes and will last a significantly longer time.
|Device|Purpose|Quantity|Cost|Vendor|
|-|:-:|-:|-:|-:|
|ESP32 Heltec board + USB-cable (WiFi/LoRa/BTE)|Chosen board for implementing the project and attaching sensors|1|~330 SEK|az-delivery.de (not in stock)
| Capacitive Soil Moisture Sensors|Reading moisture values|3|~100 SEK|[az-delivery.de](https://www.az-delivery.de/en/products/bodenfeuchte-sensor-modul-v1-2)
| Raspberry Pi 4 (+ Power supply, SD-card, case)|Hosting the docker applications|1|~900 SEK|[Electrokit.com](https://www.electrokit.com/en/product/raspberry-pi-4-model-b-4gb/)
|Powerbank with suitable output (5V)|Powering your device away from PC|1|Found in drawers |
|Soldering|A long-term solution to using a breadboard||N/A|Find a local makerspace (Might need your own wire)
|Jumper wires|Connecting sensors|~|Found in drawers|
| Breadboard (Optional)|If you decide you don't want to solder|1|Found in drawers
## Computer setup
###### *Figure 1: Devices and containers*
This project heavily relies on docker containers and MQTT for communication with the ESP32 microcontroller. Docker provides a very convenient setup if you simply want to try something before committing, as Docker Compose can be used independently of a Raspberry Pi initially by simply installing the Docker extension in VSC. This is the method that will be used during development for testing before moving the docker compose setup over to the Pi as a more permanent solution after project completion.
### All software tools, IDEs, and extensions used
:::info
All steps were performed on a PC with Kubuntu v.22.04 installed, and the RPi with Raspbian)
:::
| Name | Purpose | Type | Installation |
| -------- | -------- | -------- | ------|
|Visual Studio Code|Setup of docker containers|IDE|[Microsoft](https://code.visualstudio.com/)|
|Atom, Node.js, PyMakr|Programming the microcontroller|IDE|[LNU-IoT tutorial](https://hackmd.io/@lnu-iot/SydH7MTcw)
|Docker|Container solution|VSC extension|[VS Marketplace](https://marketplace.visualstudio.com/)
|Docker compose|Simple and portable setup of the used containers|Docker container|[dockerhub](https://hub.docker.com/)
|Home Assistant|Enables integration of different types of IoT communication protocols and devices to form a central hub for homes|Docker container| [dockerhub](https://hub.docker.com/)
|Eclipse-Mosquitto|MQTT broker|Docker container|[dockerhub](https://hub.docker.com/)
|MongoDB|Lightweight document database|Docker container|[dockerhub](https://hub.docker.com/)
|Paho-client (`httpserver.py`)|Http server in Python, responsible for collecting communication from board|Docker container|Self-written
|Bokeh charts (`datavisualization.py`)|Http server in Python with the Bokeh library for tracking moisture levels over time with data visualization|Docker container|Self-written
#### Libraries used in the code
| Name | Purpose | Used in component |
| -------- | -------- | -------- |
|[ssd1306.py](https://github.com/micropython/micropython/blob/master/drivers/display/ssd1306.py)|MicroPython OLED driver for the display (optional if you want to use the display, I merely tested it)|Heltec board
|[umqttsimple.py](https://github.com/RuiSantosdotme/ESP-MicroPython/blob/master/code/MQTT/umqttsimple.py)|MQTT MicroPython library|Heltec board
|[paho-mqtt](https://pypi.org/project/paho-mqtt/)|MQTT Python library|`httpserver.py`
|[pymongo](https://pypi.org/project/pymongo/)|Python library for interacting with MongoDB|`httpserver.py`
|[telebot](https://pypi.org/project/telebot/)|Python library for sending bot messages to Telegram|`httpserver.py`
|[pyTelegramBotAPI](https://pypi.org/project/pyTelegramBotAPI/)|Supplement to telebot|`httpserver.py`
|[requests](https://pypi.org/project/requests/)|Sending http requests |`datavisualization.py`
|[bokeh](https://pypi.org/project/bokeh/)|Drawing charts with information gathered from API|`datavisualization.py`
### Getting started
First off, install the required IDEs and extensions, as well as the libraries into their respective components. You may be able to get away with merely using VSC along with the PyMakr plugin if you want to keep the installations to a minimum, but I ran into some issues with the latest versions of the plugin so it may not work out of the box for you. Otherwise, go ahead and follow the provided tutorials on how to [install Atom](https://hackmd.io/@lnu-iot/SydH7MTcw) and how to [update your board's firmware as well as how to upload your code](https://hackmd.io/@lnu-iot/By5ZUqvOq). After this the next step is to install the Docker extension in VSC.


Now you should get familiarized with Docker Compose.

1. [Install docker along with docker-compose](https://docs.docker.com/engine/install/ubuntu/)
3. Create a directory where you want to store your project files
4. Pull the images you want to use
5. Create a file in the directory named docker-compose.yaml, read [the documentation at Step 3](https://docs.docker.com/compose/gettingstarted/#step-3-define-services-in-a-compose-file) for what the contents of this file will be. For me it's going to be something like this:

version: "2.1"
services:
  eclipse-mosquitto:
    image: eclipse-mosquitto
    container_name: MQTT-broker
    environment:
      ... secrets here ...
    volumes:
      - ./mosquitto:/mosquitto
    ports:
      - 1883:1883 
    restart: unless-stopped

  ... the rest of the services ...

Your options for each service will differ depending on what that specific application requires (environment, ports, etc ), as well as the actual configuration files for the different applications. For instance, your MQTT broker config file (Generally under Files-> `mosquitto\config\mosquitto.conf`) will require you to set a port listener. Check the documentation of the individual application/image for more details.

This file is also where you will define your environment variables for the docker containers, I strongly advise you make use of these instead of hardcoding usernames, passwords, and IP-adresses into your code. Proper use of environment variables will make it a lot easier to maintain your code. For instance, if you happen to change your database password, you only need to change this once in your docker-compose.yaml for the `httpserver.py` service instead of everywhere it's referenced in the actual code of that file. Another benefit is that you can directly reference another container within the same docker-compose to get its IP-adress, which is very useful in case it gets reassigned:

`docker-compose.yaml`

`httpserver.py`

Get your docker-compose up and running:

`docker-compose up -d`

At this point you should see something like this in VSC when you click the docker tab:


You might be missing some containers such as datavisualization and httpserver, these are the ones you need to create yourself. You will get an explanation of the contents of these files later on, as well as the repository where you can view all the code for them.

##### TODO-list for MQTT-specific behaviour:

:::info
There are visual explanations in figures 1 and 4 for this behavior, view those for clarification before implementing anything
:::
`main.py` (Heltec board):
- [x] Subscribe callback to a topic that indicates when the httpserver wants new values
- [x] Publish methods that send values, are called within the subscribe callback depending on the parameters received (topic and payload)

`httpserver.py`:
- [x] Subscribe callback to a topic that indicates when the Heltec board has sent new values
- [x] Publish method that sends requests for new values (on a timer)
##### TODO-list for other general behaviours:
- [x] Establishing a connection to MongoDB in `httpserver.py` and defining relevant methods
- [x] Creating an endpoint in `httpserver.py` that displays the x latest values in MongoDB
- [x] Defining methods and conditions in `httpserver.py` for when the telebot sends messages
- [x] Defining methods for generating Bokeh charts from the values served by `httpserver.py` and displaying these at an endpoint in `datavisualization.py`
- [x] Creating Dockerfiles for `httpserver.py` and `datavisualization.py` to build the images and include them in your docker-compose
- [x] Calibrate your sensor readings by printing which outputs you get with a completely dry sensors, as well as fully submerged in water, and write your code accordingly. You can choose to do this either directly on the board or pass along the raw value to `httpserver.py` and do your number transformations there before putting it into MongoDB.

### Flowchart of Heltec/httpserver communication in finalized project

###### *Figure 2: Communication flowchart*
### Final step: Migrate to RPi
If you don't want to host everything on your PC indefinitely, you should migrate your setup to the RPi where you have [Docker installed](https://docs.docker.com/engine/install/debian/), and then re-run your docker-compose there.

For this to work you should have created a Dockerfile for both `datavisualization.py` and `httpserver.py` (the ones you wrote yourself) so that you can rebuild the images on the RPi.

A sample script that builds both images according to the relative folder path of my project is included in the [uploaded repository](https://gitlab.com/EssR/iot2022), you can use this repository for inspiration or as a template if you wish.

## Putting everything together
###### *Figure 3: Circuit diagram*

:::info
Black: GND
Red: VCC
Yellow: Analogue output
:::
Good resources to use when connecting everything is the [pinout diagram](https://raw.githubusercontent.com/H-Ryan/Heltec/main/PyCom%20MicroPython/heltec_pinout.pdf) for the Heltec device as well as the specifications for your sensors. Take the product description and the specs of the capacitive soil moisture sensors with a grain of salt as these particular sensors are known to vary greatly across versions (even v1.2 has different versions) and depending on which one you've come across you may have missing voltage regulators or resistor connections or other issues. [This](https://www.youtube.com/watch?v=IGP38bz-K48) is a useful guide for these particular sensors.

Initially I wanted to use a breadboard, but ended up doing a bit of soldering instead as there wasn't so much to do anyway, I currently only use the one sensors.

I used VCC ➜ 5.0V as this was the only choice with my version due to aforementioned issues. Pin 13 was used for the analogue output from the sensor.

I also decided to make use of what I already had and powered the setup with a powerbank (5V) since the board has a voltage regulator on the micro USB-interface. If you decide to take this route, make sure you get one that won't shut down if the power draw isn't great enough, as this tends to be a common issue. In any case, this is a temporary solution until I decide to get something more permanent.

The power draw of the board is something around 200mA when using only Wi-Fi, as is usually the case for ESP32-boards, and I might use the OLED display in the future which would tack on another 30mA or so. The capacity of the power bank is 20000 mAh. Assuming that the power bank has seen better days I'll account for 80% of its capacity. Taking into account that I will make no use of deep sleep will have the power bank last me approximately:
:::info
$uptime = {0.8~·~20000 mAh \over 230 mA} = 69h~33m$
:::
This calculation of battery capacity over load current gives a battery life of approximately 70 hours. Decent but not completely ideal, I will have to look into optimizing this in the future with more use of deep sleep and other forms of connectivity like LoRa.

## Platform

This project aims to be entirely self-hosted rather than cloud-based, although there are arguments to be made for both. For instance, redundancy, logging, and backups are your responsibilty if you decide to take the self-hosted route. The upside is that you will have greater control over your own data and full ownership with a fully transparent overview of the usage of it, and an added bonus of no subscription fees once you get past the initial cost of purchasing your own hardware.

The platform of choice for this project is a docker composition of a pick-and-mix of different applications. Keep is mind that frequent write cycles are a problematic aspect of some of the applications used in this project, which will decrease the longevity of the SD-card used in the Pi. If you have the resources, consider storing this on something that won't be affected by these frequent write cycles, like an SSD if your RPi model supports USB boot.

The chosen applications were selected based on needs and best compatibility with my existing setup at home (Home Assistant on an RPi). Many of the components used in the setup are optional and not a requirement to get a working alerts system. At the very least, you could get away with using merely an MQTT broker and Home Assistant (Which already has an integration for a [Telegram bot](https://www.home-assistant.io/integrations/telegram_bot/)). In this instance, Home Assistant would take over the request publishing responsibility of the httpserver, and the remaining containers would simply not be used. Less fancy stuff in this setup but it will work great depending on your needs, and you wouldn't need to worry as much about the frequent write cycles.

Grafana was also tried initially as a data visualization solution, but ultimately I was more interested in personally setting up the data visualization with Bokeh charts because I already had a dashboard where this could be used (Lovelace in Home Assistant), and thus avoided having too many components to keep track of. I also felt like I'd learn more creating my own charts. Bokeh charts works very well with dictionaries and JSON data which means that querying MongoDB and the API needs very little data polishing to get a working chart. The matplotlib python library was also used and worked much like Bokeh, but Bokeh won due to vanity as it produced prettier charts with better customizability.

To summarize, my platform consists of MongoDB for data storage, Bokeh charts for data visualization, Home Assistant for QoL and beautiful dashboards to display my data (Lovelace), and a Python httpserver used as an API between the database and other components.

## The code

#### Heltec
`boot.py `

try:
    wlan.connect(ssid=envvars.WIFI_NAME, auth=(WLAN.WPA2, envvars.WIFI_PASSWORD), timeout=7000)
    while not wlan.isconnected():
        machine.idle()
    print("WiFi connected successfully")
except Exception:
    # TODO: add functionality for LoRa
    print("WiFi connection unsuccessful")

The code used for setting up the Wi-Fi connection is attempted within a try-block in case the network is not within range or cannot be used for whatever reason. In case of an exception, functionality for connecting via a new type of network can be implemented. The Heltec board is able to use LoRa and I have already set a fallback precisely for this in my exception block, but alas I am not at my usual adress at the moment. Wi-Fi it is for now. If all else fails, and it gets to the point where the MQTT connection can't be established due to no network or a connection lost, the machine resets to reconnect to the network.

`main.py`

for topic in envvars.TOPIC_SUB: # array with all current topics from import
    client.subscribe(topic)
        
def sub_cb(topic, msg):
    topic = topic.decode()
    msg = msg.decode()

    if msg == "soilmoisture":
        print("request: soil moisture readings")
        publish_pin13()

    elif msg == "temperature": # does not exist yet
        print("request: temperature readings")
        publish_pin14()

    elif msg == "humidity": # does not exist yet
        print("request: humidity readings")
        publish_pin15()
        
        # .. and all other outcomes ...

    else:
        print("request: not defined")
        publish_errorlog(topic, msg)

try:
    client = connect_and_subscribe()
except OSError as e:
    restart_and_reconnect()

while True:
    try:
        client.check_msg()
    except OSError as e:
        restart_and_reconnect()

The MQTT communication using the `umqttsimple.py` library is what makes everything happen on the board, values from the sensor are only read when the client (httpserver) requests them by triggering the subscription callback method, after which the requested values are published depending on the topic.

In my setup I have already predefined some methods for new sensors on the Heltec that I might use in the future, with corresponding request methods for them in `httpserver.py`, and the reason for this is explained a bit more in detail in Figure 5.

If for whatever reason the board is unable to check for new MQTT messages, it will assume connectivity has been lost and restart the network connection to make sure that the following messages are intercepted.

#### Containers

`httpserver.py`

# telebot, her name is Gunhild :)
gunhild = telebot.TeleBot(telegramToken)

def gunhild_alert(waterLevel):
    gunhild.send_message(chat_id, f"Vattna krukan! Fuktnivå är {waterLevel} \U0001F4A6 \U0001FAB4")

# mongoDB
uri = f"mongodb://{username}:{password}@{dbUrl}:{dbPort}/?authMechanism=DEFAULT"
client = MongoClient(uri)
db = client.moisture
collection = db.measurements

mqttc = mqtt.Client()                #
mqttc.on_connect = on_connect        #
mqttc.on_message = on_message        #
mqttc.on_subscribe = on_subscribe    # assign callback functions to prevent blocking
mqttc.connect(mqttUrl, 1883, 60)     #
mqttc.subscribe("heltec/sensor/soilmoisture", 0)
mqttc.loop_start()                   #
mqttThread = threading.Thread(target=client_request) # create a new thread for publishing periodic requests to prevent blocking

The libraries used here are telebot, Paho, and pymongo. Telebot is a Python library for connecting to Telegram's bot API and sending messages, which in my case are triggered when incoming moisture readings are below a certain threshold.

Paho is a Python MQTT library, which makes it possible for me to have MQTT communication between my httpserver and the Heltec board, this is what I use to save values to my MongoDB with the help of the pymongo library.

`datavisualization.py`

# Endpoints 
class requestHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.send_header('content-type', 'text/html')
        self.end_headers()
        bokehhtml = chart_maker() # the returned html string displayed in browser
        output = ''
        output += bokehhtml
        self.wfile.write(output.encode()) 

The httpserver API provides values to datavisualization from the MongoDB and the Bokeh library is what is used to generate charts from the values. The charts are then returned as html from the called method and served as an in-browser dashboard. This chart is also incorporated into Home Assistant's Lovelace dashboard to keep everything neat and tidy.
:::success
All of my code is [uploaded and fully commented on Github](https://gitlab.com/EssR/iot2022) with explanations if you want to try this out for yourself.
:::
## Transmitting the data / connectivity
My choice of wireless protocol for this project is Wi-Fi, although there is an implementation for a LoRa fallback (since the Heltec board supports it, might as well) for when I'm back at my usual place of residence.

The transport protocol is MQTT where the communication follows a Pull-pattern set on a timer (every 30m) from the httpserver to the Heltec board. There is a very simple reason this is used instead of a simple Push-pattern every 30m from the board: The httpserver needs to be in charge of behaviour changes.

###### *Figure 4: MQTT communication*

By letting the httpserver be in charge of requesting values, I can change the behavior of the information received without ever needing to reflash the board. I can simply define all the different methods for what can be requested from the board with regards to which sensors I'm using, and I let the httpserver be in charge of deciding when and how to receive these. I could even predefine methods on the board for pins not currently used but might be used at a later date. For me, it simply makes more sense to let the httpserver be in charge of behaviour changes as my needs are surely bound to change in the future. For instance, here are some things that I might want to change:

- The frequency of values received
- Which sensors I *currently* want values from
- Conditions and triggers on when values will be saved or ignored

Essentially I'm delegating as much behavior as I can to the httpserver for convenience. It's easier to rework containers than it is to disconnect all sensors from the microcontroller and connect it to my PC for a reflash.

The quality of service (QoS) of the message delivery is set to 0 (at most once) because missing some packets is not a big deal, regularity is not a critical aspect of this project, and the benefit of using this setting greatly outweighs the drawback.

`main.py:`

client.publish(topic_pub, value, retain=False, qos=0)

Since my board will be powered away from my PC, I want to keep power draw conservative wherever I can so that the board can run a good while without interference. [Toldinas et al. (2019)](https://ieeexplore.ieee.org/abstract/document/8765692) conducted a case study where the energy consumption of different QoS settings were compared, and their setup was similiar to mine with a Raspberry Pi as a broker. They found that QoS 0 consumes 87% less energy (Microcontroller + lithium battery) than QoS 2.

Since the frequency of the sent client requests from the httpserver (every 30m) will eliminate any issues caused by missing packets, it makes a lot of sense to use QoS 0 for increased battery life. In an extreme I would receive only one value in a day (very unlikely), and given that I mostly keep low-maintenace succulents, this scenario is perfectly fine. In conclusion, QoS setting 0 is ideal.

## Presenting the data

When the httpserver receives readings from the board, it will *always* save these to my MongoDB instance. MongoDB is a very lightweight document database and it makes perfect sense to keep values indefinitely for future analysis. It costs very little; the average document size in my db collection is 76 bytes, which means that saving values every 30m (2 values per hour) for a full year, assuming that no packets are lost, and for a single sensor results in:

:::info
$storage = {2 · 24 · 0.000076~MB · 365 } = 1.33152~MB$
:::

A little more than a MB yearly. Negligible! The saved values are presented in `httpserver.py` for other applications to use as an API:


Which is what `datavisualization.py` is for. Here is where Bokeh is used to create charts for presenting the data in a format that conveys a little more information, these charts can be created as html output which also integrates very well with Home Assistant and enables me to present my charts in the Lovelace dashboard:


## Finalizing the design


All in all it ended up being a fairly simple project hardware-wise with more focus on the software-side of things. It was fun to try new things and actually see the project to completion.

Some things I am realizing afterward is that it might be worth to give Grafana another chance, as it's also capable of getting integrated into Home Assistant, and produced very pretty charts without much fuss.

I also wanted to give LoRa a bigger chance during the project and probably could've allocated more time there even if I would've needed to redo it once I was home again.

Something I'm interested in seeing once I'm home again and have access to more plants and pots, is how well this scales, and how much I'll need to resolder or supplement with additional hardware. Or perhaps I already have everything I need!