# IoT Lab 1
###### tags: `SRM2020`
# 1.- MQTT basics
For the experiments of this part you will use an MQTT client that can be used in any OS, it's called MQTT-explorer:
You have to install it from here: http://mqtt-explorer.com
## Connecting to a public broker
There are various public MQTT brokers, for example:
* broker.hivemq.com
* broker.mqttdashboard.com
* test.mosquitto.org
In this session we will use the one offered by HiveMQ (broker.hivemq.com). Fill in the data as indicated below:
Then click on ```ADVANCED```, and add a topic, for example `test/#`:
if you want you can assing your own client ID. Now click on ```BACK```, and then on ```CONNECT```. You will start seeing something like this:
## Some basic exercise
Let's start with an easy one. Click on the ```DISCONNECT``` button. Now add the topic ``i/LOVE/Python``, then CONNECT once again.
Now, (1) write the same **identical** text for the topic (i.e., ``i/LOVE/Python``) **in the ``topic`` field of the Publish section**, select the ```raw``` option and write a text in the box below. **The text can be whatever you want**. When you are done click on the ``Publish`` button.
You'll get something like the image below... **plus all the messages written by all the other clients.** With just one publish action you actually reached various devices!!
:::danger
* What happen if you write ``i/love/Python`` instead (be careful with upper/lower cases?
* How would you build a messaging app?
:::
### Simple data collection using MQTT
Let's use now MQTT Explore with the following parameters:
```
Broker: eu.thethings.network
Username: lopy2ttn
Password: ttn-account-v2.TPE7-bT_UDf5Dj4XcGpcCQ0Xkhj8n74iY-rMAyT1bWg
Topic: +/devices/+/up
```
You will start seeing something like this:
We are currently reading some LoRaWAN sensors that are sending their data to TTN... more ahead
# 2.- Getting started with IoT devices
As an example of IoT devices we will "use" some [LoPy4](https://pycom.io/product/lopy4/) with a [PySense](https://pycom.io/product/pysense/) board or a [Pytrack](https://pycom.io/product/pytrack-2-0-x/) (**They are available for possible projects**).
LoPys are based on [MicroPython](https://micropython.org/). MicroPython is a full Python compiler and runtime that runs on the bare-metal. You get an interactive prompt, called the REPL ("Read Evaluate Print Loop"), to execute commands immediately, along with the ability to run and import scripts from the built-in filesystem. The REPL has history, tab completion, auto-indent and paste mode.
More infos can be found here: https://docs.pycom.io/gettingstarted/
To connect to the LoPy's REPL we use a Python tool called **[mpy-repl-tool](https://mpy-repl-tool.readthedocs.io/en/latest/)**.
To get access to the REPL prompt:
* `python3 -m there -i`
You'll get the `>>>` REPL prompt (see below). To exit from the REPL you have to type `Ctrl+]`
The filesystem has ``/`` as the root directory and the available physical drives are accessible from here. They are:
* `/flash`: the internal flash filesystem
* `/sd`: the SD card (if it is inserted, obviously)
A typical workflow is the following:
1. Write the micropython code in your computer
1. Copy the code file or files from your computer to the LoPy, e.g.,:
```$ python3 -m there push *.py /flash```
1. Start a serial terminal and get access to the REPL:
```$ python3 -m there -i```
1. Execute the code in file `_filename_.py` on the LoPy, with:
```>>> import _filename_```
A detailed list of commands can be found here: [mpy-repl-tool.readthedocs](http://mpy-repl-tool.readthedocs.io/en/latest/commandline.html)
Other, more friedly tools are available
Like Atom:
or Microsoft’s Visual Studio Code
### Example: accessing the sensors of the Pysense board
The PySense offers various embedded sensors, namely:
* Ambient light sensor
* Barometric pressure sensor
* Temperature sensor
* Humidity sensor
* 3 axis 12-bit accelerometer
Take a look at the code below:
```python=
from pysense import Pysense
from LIS2HH12 import LIS2HH12 # 3-Axis Accelerometer
from SI7006A20 import SI7006A20 # Humidity and Temperature Sensor
from LTR329ALS01 import LTR329ALS01 # Digital Ambient Light Sensor
from raw2lux import raw2Lux # ... additional library for the light sensor
from MPL3115A2 import MPL3115A2,ALTITUDE,PRESSURE # Barometric Pressure Sensor with Altimeter
py = Pysense()
# Digital Ambient Light Sensor
lite_s = LTR329ALS01(py)
print("Light raw (channel Blue lux, channel Red lux): " + str(lite_s.light()))
print("Light (raw2lux): " + str(raw2Lux(lite_s.light())))
# Barometric Pressure Sensor with Altimeter
bara_s = MPL3115A2(py,mode=ALTITUDE) # Returns height in meters.
barp_s = MPL3115A2(py,mode=PRESSURE) # Returns pressure in Pascals.
print("MPL3115A2 temperature: " + str(bara_s.temperature()))
print("Altitude: " + str(bara_s.altitude()))
print("Pressure: " + str(barp_s.pressure()))
# Humidity and Temperature Sensor
temp_s = SI7006A20(py)
print("Temperature: " + str(temp_s.temperature())+ " deg C and Relative Humidity: " + str(temp_s.humidity()) + " %RH")
print("Dew point: "+ str(temp_s.dew_point()) + " deg C")
# 3-Axis Accelerometer
acel_s = LIS2HH12(py)
print("Acceleration: " + str(acel_s.acceleration()))
print("Roll: " + str(acel_s.roll()))
print("Pitch: " + str(acel_s.pitch()))
# ... and also..
print("Battery voltage: " + str(py.read_battery_voltage()))
```
If we execute the code:
```
$ python3 -m there -i
>>> import reads
```
we will get something like this:
```
Light raw (channel Blue lux, channel Red lux): (0, 0)
Light (raw2lux): -1
MPL3115A2 temperature: 30.0
Altitude: 25097.87
Pressure: 100391.5
Temperature: 32.3763 deg C and Relative Humidity: 32.24997 %RH
Dew point: 13.68292 deg C
Acceleration: (0.0, 0.0, 0.0)
Roll: 2.447932
Pitch: -1.144166
Battery voltage: 4.631505
```
# 3.- A LoRaWAN networks server: the **The Things Networks**
LoRaWAN is based on the architecture below:
In this block we will review the process required to send sensors data over LoRaWAN to a specific Network server, the **The Things Networks**.
## Setting up an account
You have to first ["Create an account"](https://account.thethingsnetwork.org/register), and then you can manage your applications and devices via [The Things Network Console](https://console.thethingsnetwork.org/).
### The "GATEWAYS" section
### The "APPLICATIONS" section
To add an Application, the steps are the following
1. In the [Console](https://console.thethingsnetwork.org/), click [add application](https://console.thethingsnetwork.org/applications/add)
* For **Application ID**, choose a unique ID of lower case, alphanumeric characters and nonconsecutive `-` and `_` (e.g., `hi-world`).
* For **Description**, enter anything you like (e.g. `Hi, World!`).
2. Click **Add application** to finish.
You will be redirected to the newly added application, where you can find the generated **Application EUI** and default **Access Key** which we'll need later.
### Register the Device
The Things Network supports the two LoRaWAN mechanisms to register devices: Over The Air Activation (OTAA) and Activation By Personalization (ABP).
1. On the Application screen, scroll down to the **Devices** box and click on **register device**.
* As **Device ID**, choose a unique ID (for this application) of lower case, alphanumeric characters and nonconsecutive `-` and `_` (e.g., `my-device1`).
* As **Device EUI**, you have to use the specific value for the device.
2. Click **Register**.
You will be redirected to the newly registered device.
3. On the device screen, select **Settings** from the top right menu.
* You can give your device a description like `My first TTN device`
* Check that *Activation method* is set to *OTAA*.
* Uncheck **Frame counter checks** at the bottom of the page.
> **Note:** This allows you to restart your device for development purposes without the routing services keeping track of the frame counter. This does make your application vulnerable for replay attacks, e.g. sending messages with a frame counter equal or lower than the latest received. Please do not disable it in production.
4. Click **Save** to finish.
You will be redirected to the device, where you can find the **Device Address**, **Network Session Key** and **App Session Key** that we'll need next.
## Sending data
In this step we will use the device (the LoPy plus the PySense) registered in the step before to periodically send the sensed temperature, humidity and luminosity (lux).
```python=
import binascii
import machine
import micropython
import pycom
import socket
import struct
import sys
import time
from network import LoRa
from pysense import Pysense
from SI7006A20 import SI7006A20 # Humidity and Temperature Sensor
from LTR329ALS01 import LTR329ALS01 # Digital Ambient Light Sensor
from raw2lux import raw2Lux # ... additional library for the light sensor
RED = 0xFF0000
YELLOW = 0xFFFF33
GREEN = 0x007F00
OFF = 0x000000
def flash_led_to(color=GREEN, t1=1):
pycom.rgbled(color)
time.sleep(t1)
pycom.rgbled(OFF)
THE_APP_EUI = '70B3......02AE7C'
THE_APP_KEY = '1F438669F5......D2FC6501C90F0586'
def join_lora(force_join = False):
'''Joining The Things Network '''
print('Joining TTN')
# restore previous state
if not force_join:
lora.nvram_restore()
if not lora.has_joined() or force_join == True:
# create an OTA authentication params
app_eui = binascii.unhexlify(THE_APP_EUI.replace(' ','')) # these settings can be found from TTN
app_key = binascii.unhexlify(THE_APP_KEY.replace(' ','')) # these settings can be found from TTN
# join a network using OTAA if not previously done
lora.join(activation=LoRa.OTAA, auth=(app_eui, app_key), timeout=0)
# wait until the module has joined the network
while not lora.has_joined():
time.sleep(2.5)
# saving the state
lora.nvram_save()
# returning whether the join was successful
if lora.has_joined():
flash_led_to(GREEN)
print('LoRa Joined')
return True
else:
flash_led_to(RED)
print('LoRa Not Joined')
return False
else:
return True
pycom.heartbeat(False) # Disable the heartbeat LED
# Let's see if you inserted the required data
if THE_APP_EUI == 'VOID':
print("You must set the values of your app and device first!!")
sys.exit()
# Getting the LoRa MAC
lora = LoRa(mode=LoRa.LORAWAN, region=LoRa.EU868)
print("Device LoRa MAC:", binascii.hexlify(lora.mac()))
flash_led_to(YELLOW)
# joining TTN
join_lora(True)
py = Pysense()
tempHum = SI7006A20(py)
ambientLight = LTR329ALS01(py)
while True:
# create a LoRa socket
s = socket.socket(socket.AF_LORA, socket.SOCK_RAW)
s.setsockopt(socket.SOL_LORA, socket.SO_DR, 0)
s.setblocking(True)
temperature = tempHum.temperature()
humidity = tempHum.humidity()
luxval = raw2Lux(ambientLight.light())
print("Read sensors: temp. {} hum. {} lux: {}".format(temperature, humidity, luxval))
payload = struct.pack(">fff", temperature, humidity, luxval)
s.send(payload)
flash_led_to(GREEN)
time.sleep(60)
```
Now, if we go in the "Data" section of the TTN Application, we will see something like this:
That is the data coming from the three devices that are using this application. If you click on any of the lines of the data, you'll get:
where you can find a lot of information regarding the sending of you LoRa message.
If you check the **Payload** field, you will see a sequence of bytes... and that is actually what we sent ...
To see what we actually sent, check again the file above:
```shell=python=
temperature = tempHum.temp()
humidity = tempHum.humidity()
luxval = raw2Lux(ambientLight.lux())
payload = struct.pack(">fff", temperature, humidity, luxval)
s.send(payload)
```
As you can see we are sending the values of temperature, humidity and luminosity (lux) "compressed" as a sequence of 4*3= 12 bytes (:arrow_right: ``... = struct.pack(">fff",...``).
Now, to allow TTN to interpret these sequence of bytes we have to go the the section **Payload Format** and insert some specific code to interpret the input bytes.
# 4.- MQTT with python
:::info
**All the code necessary for this Lab session is available in [ SRM:Recursos - Laboratorios/Practicas - Labs IoT - material](https://poliformat.upv.es/x/JLaFdy)** in folder code.
You can execute the code either in your computer or online:
* online. Create an account in https://repl.it
* in your computer. You must have python3 installed and the `paho-mqtt` library:
```
$ sudo pip3 install paho-mqtt
```
The documentation of the MQTT Paho API is here: https://www.eclipse.org/paho/clients/python/docs/
:::
## Programming MQTT
In this session we will use a **public broker**. There are various public brokers (also called `sandboxes`) in the Internet. For example:
* `iot.eclipse.org` (https://iot.eclipse.org/projects/sandboxes/)
* `test.mosquitto.org` (http://test.mosquitto.org/)
* `broker.hivemq.com` (https://www.hivemq.com/public-mqtt-broker/)
An extensive list is available here: https://github.com/mqtt/mqtt.github.io/wiki/public_brokers
### A simple subscriber
File: `sisub.py` cointains the code of a simple python subscriber. This code connects to a public broker and subscribes to topic `$SYS/#`.
:::danger
* Ejecuta este código e intenta entender el resultado obtenido
:::
### A simple producer
File: `sipub.py` cointains the code of a simple python producer. This code connects to a public broker and periodically publishes random values to topic `"PMtest/rndvalue"`
:::danger
* Ejecuta este codigo (en otro terminal) e intenta entender el resultado obtenido.
* Modifica `sisub.py` para poder recibir los datos enviados por `sipub.py`.
:::
#### About retained messages:
Normally, if a publisher publishes a message to a topic, and **no one is subscribed** to that topic the message is simply discarded by the broker. If you want your broker to remember the last published message, you'll have to use the ``retained`` option.
Only one message is retained per topic. The next message published on that topic replaces the retained message for that topic.
Modify the `loop_start/loop_stop` section of `sipub.py` as indicated below:
```python=
client.loop_start()
msg_to_be_sent = "__whatevertextyouwant__"
client.publish(THE_TOPIC,
payload=msg_to_be_sent,
qos=0,
retain=False)
client.loop_stop()
```
The code will now basically send just one message and then stop.
Try now the following cases, but **remember to always execute the "subscriber" AFTER the "producer"** in all cases:
:::danger
Prueba los siguentes casos:
* Publica un mensaje con la opcion de retained a "False". ¿Qué recibe el "subscriber"?
* Publica un mensaje con la opcion de retained a "True". ¿Qué recibe el "subscriber"?
* Publica varios mensajes (diferentes) con la opcion de retained a "True" antes de activar el "subscriber". ¿Qué recibe el "subscriber"?
:::
Finally, how do you remove or delete a retained message? You have to publish a blank message with the retain flag set to true. ==Try it.==
## Some final exercise
### Ex. 1
Crea grupos de dos o tres miembros. Crea una aplicación de chat muy básica, donde todos los mensajes publicados de cualquiera de los miembros sean recibidos **solo por los miembros** del grupo.
> La lectura de texto desde el teclado en Python se puede hacer usando:
> `name = input("Enter text: ")`
:::warning
Tienes que entregar el fichero con el codigo.
:::
### Ex. 2
Escribe el codigo de un subscriptor utilizando estos datos:
```
Broker: eu.thethings.network
Username: lopy2ttn
Password: ttn-account-v2.TPE7-bT_UDf5Dj4XcGpcCQ0Xkhj8n74iY-rMAyT1bWg
Topic: +/devices/+/up
```
**(1)** Imprime todo el contenido JSON que llega en cada mensaje
**(2)** Extrae desde cada mensaje del dispositivo `lopysense2` lo siguente:
- humidity
- lux
- temperature
- _**opcional**: el listado de gateways por donde ha pasado el mensaje y con que valor de rssi_
:::warning
Tienes que entregar el fichero con el codigo.
:::
:::info
Para manejar los datos JSON, Python ofrece la clase `json`.
Ejemplo para pasar de JSON a un diccionario:
```
>>> import json
>>> tj = '{"app_id": "lopy2ttn","dev_id": "lopysense2","hardware_serial": "70B3D5499269BFA7"}'
>>> d = json.loads(tj)
>>> print(d["dev_id"])
lopysense2
```
Ejemplo para pasar de JSON a un diccionario:
```
>>> import json
>>> d = {'sensorId': 'temp1', 'Value': 25}
>>> dj = json.dumps(d)
>>> print(dj)
{"sensorId": "temp1", "Value": 25}
>>> type(dj)
<class 'str'>
```
:::
:::success
Ejemplo de mensaje de TTN
```
{
"app_id": "lopy2ttn",
"dev_id": "lopysense2",
"hardware_serial": "70B3D5499269BFA7",
"port": 2,
"counter": 12019,
"payload_raw": "Qbz/OEJCrHhBybUc",
"payload_fields": {
"humidity": 48.668426513671875,
"lux": 25.21343231201172,
"temperature": 23.624618530273438
},
"metadata": {
"time": "2020-11-18T11:30:55.237000224Z",
"frequency": 868.3,
"modulation": "LORA",
"data_rate": "SF12BW125",
"airtime": 1482752000,
"coding_rate": "4/5",
"gateways": [
{
"gtw_id": "eui-b827ebfffe7fe28a",
"timestamp": 3107581196,
"time": "2020-11-18T11:30:55.204908Z",
"channel": 1,
"rssi": 1,
"snr": 11.2,
"rf_chain": 0,
"latitude": 39.48262,
"longitude": -0.34657,
"altitude": 10
},
{
"gtw_id": "eui-b827ebfffe336296",
"timestamp": 661685116,
"time": "",
"channel": 1,
"rssi": -83,
"snr": 7.5,
"rf_chain": 0
},
{
"gtw_id": "itaca",
"timestamp": 2391735892,
"time": "2020-11-18T11:30:55Z",
"channel": 0,
"rssi": -117,
"snr": -13,
"rf_chain": 0
}
]
}
}
``
:::
Sign in with Wallet
Connect another wallet