Home environment checking and regulating device, using Webhooks and MQTT

Author: Malika Achouhan (ma226wh)

Project Overview

This beginner-friendly tutorial is providing instructions for building your own IoT (Internet of Things) home environment checking device, collecting and displaying air quality, temperature, humidity and pressure data in a data visualisation platform and based on the data, automating your air conditioner.

The following is a diagram of how exactly this project is done. However, the integration between pybytes and datacake and the datacake plugin to node-red are not necessary for the project.

The estimated time to complete this project is about 5 to 10 hours.

:information_source: Please take into consideration the fact that it may take longer based on the previous experience one has, the errors that may occur and the speed of one's Wi-Fi connection and computer.
Note that this tutorial is only for EWPE Smart powered devices which can be controled via WiFi using the EWPE Smart app.

Objective

With the pandemic, a virus attacking mostly the lungs and humans spending most of their time indoors, people are becoming more and more aware of the importance of taking care of their respiratory system and home air quality.
This is why this device was built. To help individuals:

collect data from their environment,
visualise the data on a data visualisation platform and therefore check the air quality and environment of their home and based on that,
take actions to maintain or improve it, or
automate their device(s) to do so.

The insights this project provides are how the air and overall indoor environment quality change over time.

Material

Here you will find a list with all the needed hardware, their price and a link where you can find them and their specifications.

If you want to experiment with more networks than just Wi-Fi, it is preferred that you purchase the FiPy multipack (as a standard bundle, with the expansion board of your choice) so that you also receive the LTE-M Antenna (if you want to use the LTE network you also need to order the activated 1nce SIM card) as well as the LoRa/Sigfox Antenna. In this case, due to not having a Helium spot (for LoRa) nearby and not having the SIM card, this project uses Wi-Fi.
It must be mentioned that if you choose the Pysense 2.0 X, as an expansion board, you could also neglect the Pynode+ AirQ sensor, as the board has built-in barometric pressure, temperature, humidity and ambient light sensors.

(The reason why Pysense 2.0 X was chosen for this project is because the regular expansion board was out of stock.)

Hardware	Price and Item Link	Specification
Pycom FiPy Multipack. Standard bundle, with the Pysense 2.0 X as an expansion board.	€102.28	It includes four items (1x FiPy, 1x Pysense 2.0 X, 1x LTE-M Antenna, 1x LoRa/Sigfox Antenna). (For this project only the below four items are used.)
Pycom FiPy Development Board.	€59.40 (price if ordered without SIM card)	Micropython-programmable microcontroller (something like a micro-computer), featuring WiFi, Bluetooth, LoRa, Sigfox and dual LTE-M (CAT-M1 and NB-IoT).
Pycom Pysense 2.0 X expansion board.	€29.65	Expansion board with five built in sensors. Needed to be able to connect the microcontroller to the computer through a USB cable. It also adds capabilities to your microcontroller.
Pycom Pynode+ AirQ sensor.	€24.99	BLE 5.1 air quality, temperature, humidity and pressure sensor (featuring Dialog Semiconductor DA14531 BLE 5.1 and Bosch BME688 Air Quality sensor).
USB MICRO-B(M) to A(M) data transferring and power cable.	The prices vary, but it is very likely to find one at home.	Needed to connect your module to your computer.
	Total:	Note:
	€127,27 (multipack + the sensor) or €114,04 (only the hardware needed)	If you live outside the UK, make sure to check the import duty fee too.

:information_source: The Pynode+ AirQ sensor could be replaced with a DHT11 (digital temperature and humidity) sensor and a CCS811 (eCO2 and tVOC) sensor. However, if you choose to replace it with these two sensors, you will also need a breadboard as well as jumper wires.

This project is also doable with any esp32 development board with Wi-Fi connection.
If you want to make your device portable, you can also use a LiPo-battery, an IP67 case for your module and a pynode case for your sensor.

Computer Setup

(For this project was used a PC with Windows OS.)

If you use Pycom hardware for your project, it is recommended that you also check the guides provided by the company itself. For this part, you can refer to this guide.

To setup your computer, using MicroPython as a programming language, you will need to install the Pymakr plugin which is for both Atom IDE (integrated development environment) and Visual Studio Code. However, for this project is used the Atom IDE, because it gives more capability to the plugins and it's easy to use.

For the plugin:

install the latest (current) version of Node.js,
install Atom,
open it and go to File > Settings > Install to open the package manager,
search for pymakr and install it.

Fig. 1. Sourced from.

:thumbsup: Now your computer setup is ready! Time to move on to flashing your firmware.

First of all, you have to connect your module to your computer. To do so, insert your development board, (in this case FiPy), into your expansion board (Pysense 2.0 X). Then, to communicate with it, connect your module to the computer with the USB cable.

:warning: You have to place your board in such way, that the reset button and RGB LED are above the USB connector.
:warning:Also, be very careful and gentle when plugging and unplugging the USB cable to your expansion board, as its USB connector is very sensitive and it can be easily damaged. So, when you want to disconnect and re-connect your module to your computer, do it by accordingly unplugging or plugging the cable from and to your computer.

Pycom Firmware Update Tool.

Download and install it,
open it,
click on continue,
again click on continue,
in the port selection section, choose the COM port where you have connected your module,
in the type section, select "development",
click on continue,
in the next page (Advanced Settings), the tool should automatically detect your device information,
select "Erase during update", "CONFIG Partition" and "NVS Partition",
click on continue and then wait for your firmware to be updated.

You must be careful to not interrupt the updating procedure in any case, as this may damage your board.

:thumbsup: When it's done updating, you can click on "Done".
Now you are ready to start with your project!

To upload the code:

open Atom,
connect your device,
(you will know it is connected when you see >>> in the REPL which, in the Python programming language, means we can start typing commands.)
If you want to test if it works, you can type in a command like print ("Hello!") or 1+1 and then press Enter to see the output.
(It should look like the output in the picture below:)
In your computer, create a folder with the name of your project,
go in Atom and click on File > Open folder and select the folder you created,
(after you do this, the folder should appear in the left panel in your Atom workplace)
right-click on your folder > New File and name it as main.py,

:information_source: main.py is the start point of code execution of a project. So here, you will have to insert the code of the main function of your project.

(In this project's case, the main function is the data collection of the environment, executed by the sensor.)

right-click on your folder > New File and name it as boot.py.
(In this folder will be the code for the Wi-Fi connection).
After you write the code in the files, you need to:
- press Ctrl + S to save,
- press Ctrl + Alt + S to upload and
- press Ctrl + Alt + R to run it.

Putting Everything Together

This tutorial is relatively easy in terms of electronics. The sensor used connects to the microcontroller through BLE (bluetooth low energy).
To do that, you need to upload to your microcontoller and run this code in the main.py.
The sesnor is powered by a Panasonic CR2032 LITHIUM 3V COIN 20mm.

(Know that if we set it's sleep timer to 1 minute, with its gas sensor enabled, the power consumption will be between 220uA and 17.10mA).

Platform

After everything is put together, is time to prepare the platform where the collected data will be visualised.

For this project, I use 3 different data visualisation platforms simultaneously. That is because I:

wanted to learn how to use different data visualisation platforms (as this project was made as part of the Introduction to Applied IoT course and I was a complete beginner),
decided to make my project more complex, control my ac and see how these platforms interact with one another,
wanted to avoid opening a port in my router and having my MQTT Broker available online (for the communication of the data visualisation platform with my ac, if it was done with either one of Pybytes or Datacake, or both of them) as I did not have the time to secure it.

Pycom has it's own data visualisation platform, Pybytes, which is based on a cloud.
Pybytes is a pretty straight-forward platform and easy to use.
The reason why I kept this platform in my project is because, it has some interesting features, that I will definitely try in the future. These are:

the machine learning models you can create to analyse your data,
the possibility of managing multiple devices, projects and data from the same place, and the coolest one,
the "Pymakr Online" function, from where you can manage and compile the code of your device remotely!

However, there are some disadvantages like the following ones:

while connecting your device to the platform, you may encounter some issues, as I (including some classmates) did.
Personally, the issue was that I could not connect my device to Pybytes through Wi-Fi.
To solve that, I tried to connect them offline through the Pycom Firmware Update Tool. So, I double-checked my Wi-Fi credentials in the code on Atom, as well as on Pybytes (Settings> Networks> Wi-Fi Credentials), I repeated the offline firmware update procedure, with the activation token, several times and I also deleted my device and followed the whole process of adding it again (it took me more than 2 hours until I connect it).

:information_source: If you are sure that you did everything right but it still does not work, then it may be a bug in the platform.

The data visualisation isn't as advanced as it could or should be.
In the charts, you can only see the data for the past 24 hours and in the signals, you can only see the data of the past couple of days (depends on how often you have programmed your device to send the data and therefore the amount of data received in pybytes).
The guides provided by Pycom, seem to be a little old, since some things do not work in the same way as shown there, the platform design and the place where things are at, seem to have changed a bit.
For instance, the signals are set automatically, according to those sent by the device (you control that from the code on Atom).
When trying to integrate Pybytes to Datacake, using Webhook as a transport protocol, Pybytes would not send the test message, nor the data to datacake.
The url provided by datacake was working because when it was tested with Postman it would receive the messages. So to make it work, I repeated several times the process of deleting the webhook as well as my device from both platforms, re-adding it to both platforms and re-creating the webhook in Pybytes.

:information_source: In general, it is a good platform for beginners but, there is the possibility that it causes a struggle when adding and connecting your device.
Also, you will need to browse around the platform, so you figure out where things are at.

After learning how to use Pybytes, I wanted to learn how to use a more advance platform, Datacake by also using MQTT (MQ Telemetry Transport).

Since I wanted to keep using Pybytes for my project, I decided to integrate Pybytes with Datacake.

After doing so, I chose to keep it for my project because Datacake:

is a cloud-based data visualisation platform,
offers a free subscription for up to two devices,
each device automatically outputs data via their MQTT Broker,
in comparison to pybytes, has more advanced data visualisation, as it:
- offers more choices in terms of widgets and charts,
- has available older data in history,
- offers the possibility to automate notifications (create rules) based on the received data.

The last data visualisation platform I used, is Node-red which is free and based on local installation.

Node-red kind of looks like Scratch, but is more advanced, compared to the previous two platforms.
The reason I wanted to use it in my project is because, it provides me with the ability to set specific orders for my a/c. So what I did is basically program my a/c to turn on or off and to certain modes, after the signals received reach certain values.

I also set up a Mosquitto MQTT Broker, following this tutorial for automating my a/c. I followed this tutorial but I replaced Docker with Node-Red.

The Code

In this section, you will find the core functions of my code, what they do and where I use them.

Code for Wi-Fi connectivity





















def do_connect():
    from network import WLAN
    import time
    import pycom
    import machine
    pycom.wifi_mode_on_boot(WLAN.STA)   # choose station mode on boot
    wlan = WLAN() # get current object, without changing the mode
    # Set STA on soft rest
    if machine.reset_cause() != machine.SOFT_RESET:
        wlan.init(mode=WLAN.STA)        # Put modem on Station mode
    if not wlan.isconnected():          # Check if already connected
        print("Connecting to WiFi...")
        # Connect with your WiFi Credential
        wlan.connect('Wi-Fi SSID', auth=(WLAN.WPA2, 'Password'))
        # Check if it is connected otherwise wait
        while not wlan.isconnected():
            pass
    print("Connected to Wifi")
    time.sleep_ms(500)
    # Print the IP assigned by router
    print('network config:', wlan.ifconfig(id=0))

This code is to be copied and pasted in the boot.py file.

:information_source: Remember to replace the Wi-Fi SSID and the Password with the credentials of your Wi-Fi Network.
This file will run before the main.py, so if you want to connect to other networks too, you should define it here.
You can find more Wi-Fi code examples here.

BLE Connection between the sensor and the board










from network import Bluetooth
import time
import ubinascii
import struct
import math
import pycom

bt = Bluetooth()
bt.stop_scan()
bt.start_scan(-1)

This code is to be copied and pasted in the main.py.
(The full code for the sensor can be found here).

Send data to pybytes







































while True:
    adv = bt.get_adv()

    if adv: # and adv.rssi>-80:# and ubinascii.hexlify(adv.mac)==b'cd9e13c0f24a':
        read_adv = bt.resolve_adv_data(adv.data, Bluetooth.ADV_MANUFACTURER_DATA)
        if read_adv==None:

            pass
        else:
            manuf = ubinascii.hexlify(read_adv)
            manuf_data = ubinascii.hexlify(read_adv[0:4])
            # print(manuf_data)
            if (manuf_data == b'4c000215') :#or (manuf_data == b'd2000215')):# company id=d2 is Dialog, b'4c000215' is Apple's id and it implies ibeacon
                # print("mac:", ubinascii.hexlify(adv.mac))
                uuid_raw = read_adv[4:20]
                uuid = ubinascii.hexlify(uuid_raw)
                name, air, press =byte_to_info(uuid_raw)
                if name == "PyN":
                    print("rssi:",adv.rssi)
                    major = ubinascii.hexlify(read_adv[20:22])
                    minor = ubinascii.hexlify(read_adv[22:24])
                    tx_power = ubinascii.hexlify(read_adv[24:25])
                    tx_power_real = twoscmp(int(tx_power, 16))
                    major_int = int(major, 16)
                    major_f = major_int/100 # bme688
                    minor_int = int(minor,16)
                    minor_f = minor_int/100 # bme688, it is divided by 10 initially in the dialog's firmware.
                    print("Temperature: {} C, Humidity: {} %r.H.".format(major_f, minor_f), time.time())
                    air_score = air_quality_score(minor_f, air)
                    pybytes.send_signal(0, major_f)
                    pybytes.send_signal(1, minor_f)
                    pybytes.send_signal(2, air_score+0.01)
                    pybytes.send_signal(3, press)
                    print("air {}".format(air_score))
                    print("sent to pybytes")
                    time.sleep(3600)
                    print("done sleeping")
    else:
        time.sleep(1)

This while loop is for sending the signals to pybytes, as well as setting the time sleep for the sensor (the value in the parenthesis, in line 36, is the number of seconds the sensor does not sense nor send data).
Note that signal:

is temperature,
is humidity,
is air quality/score,
is pressure.

This should be copy and pasted in the main.py file.

:information_source: In the code, in line 32, I've set the signal 2 as air_score+0.01 because when the air quality is good (therefore equal to 0), I got a "-" in the signal on pybytes. So, I did this, to be sure that the code works and the sensor sends data, but at the same time, not change the value of this signal.

Code Libraries

I have also used this (ewpe-smart-mqtt) library for setting an MQTT bridge for my EWPE Smart powered a/c (which can be controled via WiFi using EWPE Smart app).
To be able to use the above-mentioned library, you will first need to set up an MQTT server (I personnaly chose Mosquitto and to do that, I followed this tutorial).
To be able to automate my a/c, I also integrated Datacake with Node-Red using this plugin.

Transmitting the Data / Connectivity

As I mentioned before I use Pybytes and Datacake, so the data is transmitted to the internet. I also use Node-red, so the data is transmitted back to the local server.

I chose to send the data every one hour, because where I live summers are very hot and winters are very cold, so the changes in temperature can be quick and big.

The wireless protocol I chose to use, for the data transmission, is Wi-Fi because:

it transmits wide range of data quickly,
I do not have a Helium spot for LoRa nearby and
the 1nce SIM card would be late to be activated and arrive.

However, compared to LoRa, Wi-Fi is much more energy consuming.

In my project, I use two different types of transport protocols:

Webhook, in order to integrate Pybytes to Datacake and
MQTT, to connect Node-red to Mosquitto and the later to the MQTT to a/c Bridge.

As I have mentioned in the Platform section, I did not want to open a port in my router, due to safety reasons, and that is why I chose to use these three platforms and open a port in my personal network, so that my MQTT Broker is not available online.
But, because of my data going outside my network and then coming back in, it takes some time until they arrive to node-red and therefore order my a/c.
Also, this increases the battery consumption, as it needs to be functioning the whole day and it uses Wi-Fi.

Presenting the Data

Here I will talk about all three platforms separately.

Pybytes:

to build a dashboard, after having connected your device, you need to:
- go to Devices> select your device> Dashboard> Add Widget
In the dashboard charts, the data is preserved for only 24 hours. However, in the signals, if you click on the signal whose data you are interested in, you can see some older data too, according to the amount of data sent.
This is how my dashboard looks like:
The data is saved on the database by the frequency it is sent. In our case, that is every one hour.
The reasons I chose this database, are the learning purposes and the nice features it has.

The data from Pybytes (through a Webhook integration) is transmitted to Datacake.

Datacake

To build a dashboard, you need to:
- go to Devices> select your device> Configuration> scroll down to "Fields" and edit your fields,
- then go to Devices> select your device> Dashboard> click on the top right switch> click on "Add Widget" edit them how you want and when you are done, turn off the switch.

The followings are views of my dashboard:

The data is saved on the database by the frequency it is sent. In our case, that is every one hour, just like in Pybytes.
The data of free accounts is preserved for one week.
I chose this database as it is more advanced and it also offers the possibility to create triggers.
The trigger, I created in this platform, is a rule that sends me an email every time the sensor senses high levels of humidity.
(I would use a smart plug for that to be able to automate my dehumidifier, but I did not have the time to do that. I will try it in the future.)

Lastly, I integrated Datacake to Node-Red using the node-red-contrib-datacake plugin.

Node-Red

Node-Red looks like Scratch. It is not as much of a data visualisation platform as it is for giving orders. However, you can see the data received from datacake in the right-side panel of your flow, by clicking on the debug button (the one that looks like a bug).
In your flow, you create the orders you want, edit them by double clickin on them and when you are done, click on 'Deploy'.

The flows, of the commands I have done for the time being, look like:

(Flow of activating the a/c to a low temperature, when the temperature is higher than a value.)

(Flow of activating the ionizer when the air quality is low.)

(Flow of activating the a/c to a high temperature, when the temperature is lower than a value.)

The data is saved in the database by the frequency it is sent. In our case, that is every one hour, just like in Pybytes.
I chose this database because it is more advanced and for the reasons mentioned here in the Node-Red part.
As shown, in the images above, for the time being the automation created, based on the data, is to automatically activate, deactivate and turn on the ioniser mode of my a/c.

Finalising the design

To conclude, with this project, I wanted to learn as much as I could, but at the same time, do a complex project. Had I had more time, (and what I am planning to do in the future is that) I would like to:

make my device more portable by;
- using a case for my sensor and module,
use LoRa to make it more sustainable,
add my dehumidifier,
create more commands and
see what plants are actually improving it, or what plants should be added indoors in order to improve it.