# LAB 3: LoRaWAN example: send sensors values via TTN to Ubidots
> KIC, June 2019
> version LONG

This lab aims to offer you an hands-on experience with the complete process of generating data from an IoT device, send the information through a LoRaWAN gateway to the TTN server, and how to collect and visualize the data.

The overall map of what we will be using is shown in the figure below.

# Step 1: setting up the TTN (The Things Network) server

As describe before in class, in TTN there are two basic elements: the *applications* and the *devices*.

==We already set-up an application and associated to it the devices you will use in this lab. If you want to see an overview of how to do it, [click here](https://hackmd.io/QGdgS_bVT4uaiT-L7MJUfg)==

To use TTN, you need an account.
1. [Create an account](https://account.thethingsnetwork.org/register).
2. Select [Console](https://console.thethingsnetwork.org/) from the top menu.
3. Select [Applications](https://console.thethingsnetwork.org/applications). You will have not application at the moment. Give your username to us so that we can add you to the application we created.

Eventually, you will have something like this:

# Step 2: feeding the TTN server with data from the sensors

In this step we will use the LoPy plus the PySense board to periodically send the temperature, humidity and luminosity (lux).
The LoPy are already registered and are running the code available [here](https://github.com/pmanzoni/KIC2019/tree/master/lopyscode).

**When you power up your device it will automatically start generating data to TTN.**

Now, go in the "Data" section of your TTN Application. You will see something like:
The first line in the bottom is the message that represents the conection establishment and the other lines the incoming data.

If you click on any of the lines of the data, you'll get:


where you can find a lot of information regarding the sending of you LoRa message.

If you check the Payload field, you will see a sequence of bytes... Now, to allow TTN to interpret these sequence of bytes we have to go the the section **Payload Format** and insert the code in file `ttn_decode_thl.txt` as is:


**IMPORTANT: remember to click on the "save payload function" button at the bottom of this window**

==Actually, since the application is shared among all, this task has to be done simply by one of you.==

Go back to the Data window in TTN; you will now see that lines show some more infos:


and if you click on any of the lines you will see the data in readable format:

**Now, TTN does not store the incoming data for a long time. If we want to keep these data, process and visualize them, we need to get them and store them somewhere.**

TTN can be accessed using MQTT. We will first of all (**Step 3**) see how to acces the data using MQTT.

Then, (**Step 4**) we will see how to send data to a web platform.

Finally, (**Step 5**) we will collect data from TTN and send it directly to Ubidots, again using MQTT.

------

# Step 3
TTN does not store the incoming data for a long time. If we want to keep these data, process and visualize them, we need to get them and store them somewhere.

TTN can be accessed using MQTT. We will first of all write the code necessary to access TTN through MQTT and read the incoming data.

Later we will collect data from TTN and send it to Ubidots using MQTT.

> All the details of the TTN MQTT API, can be found here: https://www.thethingsnetwork.org/docs/applications/mqtt/quick-start.html

## Simple data collection using MQTT
In this Lab you will work with MQTT using the [**MQTT Paho** library](https://www.eclipse.org/paho/clients/python/) for Python. Programming will be done using a cloud development environment called [**repl.it**](https://repl.it/login). If you don't already have an account for it, you have to create one right now (it takes 3 seconds). repl.it will take care of installing the MQTT Paho library.

Once in your `repl.it` account, click on the  button and then select Python. You will see something like this:

The box on the right will show the execution of the code.
**IMPORTANT:**
* The code has to be inserted in the box indicated with the blu line in the Figure.
* When executing the code of the other examples you will have to create a "new repl"; **do not** add files in the box indicated with the red line in the Figure.

The code to be used is the one below; it is also available in the github repository named as [step3.py](https://github.com/pmanzoni/KIC2019/blob/master/step3.py).

Remember to first properly set the values for the username (`TTN_USERNAME`) which is the **Application ID** and the password (`TTN_PASSWORD`) which is the Application **Access Key**, in the bottom part of the _Overview_ section of the "Application" window.

import sys
import time
import base64

import json
import struct

import paho.mqtt.client as mqtt

THE_BROKER = "asia-se.thethings.network"
THE_TOPIC = "+/devices/+/up"

# SET HERE THE VALUES OF YOUR APP AND DEVICE:
# TTN_USERNAME is the Application ID
TTN_USERNAME = "VOID"
# TTN_PASSWORD is the Application Access Key, in the bottom part of the Overview section of the “Application” window.
TTN_PASSWORD = "VOID"

# The callback for when the client receives a CONNACK response from the server.
def on_connect(client, userdata, flags, rc):
    print("Connected to ", client._host, "port: ", client._port)
    print("Flags: ", flags, "return code: ", rc)

    # Subscribing in on_connect() means that if we lose the connection and
    # reconnect then subscriptions will be renewed.
    client.subscribe(THE_TOPIC)

# The callback for when a PUBLISH message is received from the server.
def on_message(client, userdata, msg):

    themsg = json.loads(msg.payload.decode("utf-8"))

    payload_raw = themsg["payload_raw"]
    payload_plain = base64.b64decode(payload_raw)
    vals = struct.unpack(">fff", payload_plain)

    gtw_id = themsg["metadata"]["gateways"][0]["gtw_id"]
    rssi = themsg["metadata"]["gateways"][0]["rssi"]

    print("%s, rssi=%d" % (gtw_id, rssi))
    print("@%s >> temp=%.3f hum=%.3f lux=%.3f" % (time.strftime("%H:%M:%S"), vals[0], vals[1], vals[2]))



client = mqtt.Client()

# Let's see if you inserted the required data
if TTN_USERNAME == 'VOID':
    print("You must set the values of your app and device first!!")
    sys.exit()
client.username_pw_set(TTN_USERNAME, password=TTN_PASSWORD)

client.on_connect = on_connect
client.on_message = on_message

client.connect(THE_BROKER, 1883, 60)

client.loop_forever()

Now, with this code executing, **and your device generating data to TTN (as before)** you should start seeing data coming to your `repl` console:


# Step 4: Connecting to the Ubidots platform

In this block you will experiment about how MQTT can be used to send data to a cloud based platform. This procedure allows you to store your data in a cloud based repository and to analyze your data with software tools made available by the used platform. For these experiments we will use the [Ubidots](https://ubidots.com/) plaftorm.

You will have to first create your free account in the Ubidots platform here: https://app.ubidots.com/accounts/signup/

Then you have to add a **Device** (select first the "Devices" section in the top on the web page):

and then create a "Default" type variable:

Now we will send data to our device using MQTT. Take a look first to the Ubidots MQTT API Reference: https://ubidots.com/docs/hw/#mqtt

## Sending data to Ubidots

We will use the MQTT "producer" below that can be found in the github repository named as [step4.py](https://github.com/pmanzoni/KIC2019/blob/master/step4.py), in repl.it to send data to the ``variable`` of our Ubidots ``device``:

The name of the broker for educational users is **"things.ubidots.com"**. To interact with it, you will need a TOKEN. To get yours click on “API Credentials” under your profile tab:


In my case I have:


To connect to the MQTT broker you'll have to use your ``Default Token`` as the MQTT username, and `None` as password.

The **topic** you have to use is **`/v1.6/devices/{DEVICE_LABEL}`** where you have to replace the `{DEVICE_LABEL}` with the name you gave to your device. For example `kictest` as in the figure below.


The data must be represented using JSON. The simplest format is: `{VARIABLE_NAME:10}`. In the examples above would be `{thevar:10}`.

Read the code below to understand the details and to determine the variables specific for your case.

import random
import time
import json


import paho.mqtt.client as mqtt

THE_BROKER = "things.ubidots.com"
CLIENT_ID = ""
UBI_USERNAME = "VOID"
UBI_PASSWD = ""
TOPIC = "/v1.6/devices/"
DEVICE_LABEL = "VOID"
VARIABLE_LABEL = "VOID"

# The callback for when the client receives a CONNACK response from the server.
def on_connect(client, userdata, flags, rc):
    print("Connected to ", client._host, "port: ", client._port)
    print("Flags: ", flags, "returned code: ", rc)

# The callback for when a message is published.
def on_publish(client, userdata, mid):
    print("sipub: msg published (mid={})".format(mid))


client = mqtt.Client(client_id=CLIENT_ID, 
                     clean_session=True, 
                     userdata=None, 
                     protocol=mqtt.MQTTv311, 
                     transport="tcp")

client.on_connect = on_connect
client.on_publish = on_publish

client.username_pw_set(UBI_USERNAME, password=UBI_PASSWD)
client.connect(THE_BROKER, port=1883, keepalive=60)

client.loop_start()

while True:

    # Simulates sensor values
    sensor_value = random.random() * 100

    # Builds Payload and topíc
    payload = json.dumps({VARIABLE_LABEL: sensor_value})
    topic = "{}{}".format(TOPIC, DEVICE_LABEL)

    client.publish(topic, payload, qos=1, retain=False)

    time.sleep(5)

client.loop_stop()

You'll get:


So try to repeat all the previous steps with your own device and variable.

# Step 5: Data collection using Ubidots

What we have to do now is:
1. prepare Ubidots to receive our data
2. modify the previous code to upload it to Ubidots automatically

### phase 1: preparing Ubidots
In Ubidots create a device named `step5` with the following variables names (exactly): `luxx, humidity, temperature`.


### phase 2: modify the code

Now, again using [repl.it](https://repl.it/), execute the code below that can be found in the github repository named as [step5.py](https://github.com/pmanzoni/KIC2019/blob/master/step5.py).

Remember to first properly set the vales for `TTN_USERNAME`, `TTN_PASSWORD`, and `UBIDOTS_USERNAME`.

import json
import sys
import time
import base64
import struct

import paho.mqtt.client as mqtt

TTN_BROKER = "asia-se.thethings.network"
TTN_TOPIC = "+/devices/+/up"

UBIDOTS_BROKER = "things.ubidots.com"

# SET HERE THE VALUES OF YOUR APP AND DEVICE
# TTN_USERNAME is the Application ID
TTN_USERNAME = "VOID"
# TTN_PASSWORD is the Application Access Key, in the bottom part of the Overview section of the “Application” window.
TTN_PASSWORD = "VOID"
# The User DEFAULT TOKEN
UBIDOTS_USERNAME =  "VOID"


# The callback for when the client receives a CONNACK response from the server.
def on_connect_ttn(client, userdata, flags, rc):
    print("Connected to ", client._host, "port: ", client._port)
    print("Flags: ", flags, "return code: ", rc)

    # Subscribing in on_connect() means that if we lose the connection and
    # reconnect then subscriptions will be renewed.
    client.subscribe(TTN_TOPIC)

def on_connect_ubi(client, userdata, flags, rc):
    print("Connected to ", client._host, "port: ", client._port)
    print("Flags: ", flags, "return code: ", rc)

def on_message_ttn(client, userdata, msg):

    themsg = json.loads(msg.payload.decode("utf-8"))

    payload_raw = themsg["payload_raw"]
    payload_plain = base64.b64decode(payload_raw)
    vals = struct.unpack(">fff", payload_plain)

    gtw_id = themsg["metadata"]["gateways"][0]["gtw_id"]
    rssi = themsg["metadata"]["gateways"][0]["rssi"]

    print("%s, rssi=%d" % (gtw_id, rssi))
    print("@%s >> temp=%.3f hum=%.3f lux=%.3f" % (time.strftime("%H:%M:%S"), vals[0], vals[1], vals[2]))

    # JSONining the values according to the Ubidots API indications 
    payload = {"temperature": vals[0], "humidity": vals[1], "luxx": vals[2]}

#    client_ubi.connect(UBIDOTS_BROKER, 1883, 60)
    client_ubi.loop_start()

    client_ubi.publish("/v1.6/devices/step5", json.dumps(payload))

    client_ubi.loop_stop()


client_ttn = mqtt.Client()
client_ubi = mqtt.Client()

# Let's see if you inserted the required data
if TTN_USERNAME == 'VOID':
    print("\nYou must set the values of your app and device first!!\n")
    sys.exit()
client_ttn.username_pw_set(TTN_USERNAME, password=TTN_PASSWORD)

# Let's see if you inserted the required data
if UBIDOTS_USERNAME == 'VOID':
    print("\nYou must set the values of Ubidots user first!!\n")
    sys.exit()
client_ubi.username_pw_set(UBIDOTS_USERNAME, password=None)

client_ttn.on_connect = on_connect_ttn
client_ubi.on_connect = on_connect_ubi
client_ttn.on_message = on_message_ttn

client_ttn.connect(TTN_BROKER, 1883, 60)
client_ubi.connect(UBIDOTS_BROKER, 1883, 60)

client_ttn.loop_forever()

As you can see, the code connects to the two brokers, reads from the TTN broker the incoming data, adapts the format of the data to the Ubidots API, and then publish it to the Ubidots broker.

Check in the Ubidots interface and you should see the incoming data. For example: