# MQTT

> Building a MQTT broker using a RPi, sending data via WiFi, actuation using MQTT

Basically this lab is organized so that you can have an hands-on experience with MQTT and learn how to "publish" and "subscribe" to data. To this end you will use:
1. a "sandbox" broker
1. the ThingSpeak broker
1. your own broker

Among other things you will:
1. produce data using the command line
1. produce data by code using the LoPy devices and the Raspberry Pi
1. produce data to your own channel in ThingSpeak

**All the code that you will be using is available [here](https://www.dropbox.com/sh/6wkgyh2r5ci6x1n/AAAOAW8O0x0Fk-7u6hK8zSlya?dl=0).**

## Hardware
Each group will use:
* A Raspberry Pi 3 with *Ubuntu MATE 16.04.2*
* A LoPy connected through a PySense board

The various elements are supposed to be connected as indicated in the figure below.

For the next steps you will use one or more terminals (e.g., `xterm`) connected to your Raspberry Pi via `ssh -X`.

## Installing the MQTT broker

There are various MQTT brokers that can be used with a Raspberry Pi device. A quite complete list can be found here https://github.com/mqtt/mqtt.github.io/wiki/servers

The most widely used are:
* http://mosquitto.org/
* http://www.hivemq.com/
* https://www.rabbitmq.com/mqtt.html
* http://activemq.apache.org/mqtt.html
* https://github.com/mcollina/mosca

For our experiments we will use [**Eclipse Mosquitto**](https://projects.eclipse.org/projects/technology.mosquitto). For more documentation check here: https://mosquitto.org/

### Installation steps:
To install the broker and some utilities you need to execute the following steps:

sudo apt-add-repository ppa:mosquitto-dev/mosquitto-ppa
sudo apt-get update
sudo apt-get install mosquitto mosquitto-clients

Once installed, please take a look at its [`man page`](https://mosquitto.org/man/mosquitto-8.html).

### Managing the broker

To check if the broker is running you can use the command:

sudo netstat -tanlp | grep 1883

> note: "-tanlp" stands for: tcp, all, numeric, listening, program

alternatively use:

ps -ef | grep mosquitto

To start and stop its execution use:

sudo /etc/init.d/mosquitto start/stop

if necessary, to avoid that it restarts automatically, do: `sudo stop mosquitto`

To run the broker execute:

sudo mosquitto –v

> note: "-v" stands for "verbose mode" and can be useful at the beginning to see what is going on in the broker. Can be convenient to use a dedicated terminal for the broker to execute in, if the "-v" option is used.

## Clients for testing
The broker comes with a couple of useful commands to quickly publish and subscribe to some topic. Their basic syntax is the following.

mosquitto_sub -h HOSTNAME -t TOPIC
mosquitto_pub -h HOSTNAME -t TOPIC -m MSG

More information can be found:
* https://mosquitto.org/man/mosquitto_sub-1.html
* https://mosquitto.org/man/mosquitto_pub-1.html

### Some example:
Activate the broker with `mosquitto -v` in a terminal and open 2 more terminals, more or less like this:
This way you'll control what your broker is doing.

Let's start with a easy one. In one terminal write:

mosquitto_sub -t i/LOVE/Python

the broker terminal should show something like:


the broker registered the subscription request of the new client. Now in another terminal, execute:

mosquitto_pub -t i/LOVE/Python -m "Very well!"

in the broker terminal, after the new registration messages, you'll also see something like:


meaning that the broker received the published message and that it forwarded it to the subscribed client. In the terminal where `mosquitto_sub` is executing you'll see the actual message appear.

Try now:

mosquitto_pub -t i/love/python -m "Not so well!"

**What happened? Are topics case-sensitive?**

Another useful option of `mosquitto_pub` is the following:

mosquitto_pub -t i/LOVE/Python -l

it sends messages read from stdin, splitting separate lines into separate messages. Note that blank lines won't be sent. Give it a try ... you basically obtained a MQTT based unidirectional chat channel :smile:

**qos (Quality of Service)**. Adding the `-q` option, for example to the `mosquitto_pub` you'll see the extra message that are now interchanged with the broker. For example, doing:

mosquitto_pub -t i/LOVE/Python -q 2 -m testing

you'll get:


compare this sequence of messages with the one obtanined with `-q 0` or with `-q 1`.

### Retained messages:
Normally if a publisher publishes a message to a topic, and *no one is subscribed* to that topic the message is simply discarded by the broker. If you want your broker to remember the last published message, you'll have to use the ```retain``` option. Only one message is retained per topic. The next message published on that topic replaces the retained message for that topic.
> To set the retain message flag you have to add `-r` using the Mosquitto clients.

So try the following cases, but remember now to start the subscriber **after** the publisher:
1. Publish a message with the retain message flag not set, like we did before. What happens?
1. Publish a message with the retain message flag set (`-r`). What happens?
1. Publish several (different) messages with the retain message flag set before starting the subscriber. What happens?
2. Publish a message with the retain message flag **not** set again. What happens?

Finaly, how do I remove or delete a retained message? You have to publish a blank message with the retain flag set to true which clears the retained message. Try it.

### Public brokers
There are also various public brokers in Internet, also called `sandboxes`. For example:
* `iot.eclipse.org`
* more infos at: https://iot.eclipse.org/getting-started#sandboxes
* `test.mosquitto.org`
* more infos at: http://test.mosquitto.org/
* `broker.hivemq.com`
* more infos at: http://www.hivemq.com/try-out/
* http://www.mqtt-dashboard.com/

we will always access them through port `1883`. Repeat some of the exercise above with one of these sandboxes (remember to use the `-h` option). Any difference?

## Accessing ThingSpeak via MQTT

ThingSpeak is an IoT analytics platform service that allows you to aggregate, visualize and analyze live data streams in the cloud. ThingSpeak provides instant visualizations of data posted by your devices to ThingSpeak. With the ability to execute MATLAB® code in ThingSpeak you can perform online analysis and processing of the data as it comes in. Some of the key capabilities of ThingSpeak include the ability to:

**You will see much more about this platform in a later lab session.**

### Creating a *channel*
You first have to sign in. Go to https://thingspeak.com/users/sign_up and create your own account. Then you can create your first channel. Like for example:

In the "Private View" section you can get an overview of your data:

Take a look to the other sections. To connect to your channel, you need the data in the API Keys section. In my case it says:

Now, ThingSpeak offers either a REST and a MQTT API to work with channels. See here: https://es.mathworks.com/help/thingspeak/channels-and-charts-api.html
Unfortunately, for the moment, ThingSpeak supports only **publishing** to channels using MQTT.

### Exercise
Anyway, let's publish to your channel field feed some random value using `mosquitto_pub`.


Consider that:
1. the hostname of the ThinSpeak MQTT service is "mqtt.thingspeak.com"
2. the topic you have to use is `channels/<channelID>/publish/fields/field<fieldnumber>/<apikey>` where you have to replace:
* <channelID> with the channel ID,
* <fieldnumber> with field number that you want to update, and
* <apikey> with the write API key of the channel.
3. finally, remember that ThingSpeak requires you to:
* set the PUBLISH messages to a QoS value of 0.
* set the connection RETAIN flag to 0 (False).
* set the connection CleanSession flag to 1 (True).
4. more infos here
https://es.mathworks.com/help/thingspeak/publishtoachannelfieldfeed.html

# MQTT clients with MicroPython and the LoPy

## First, some basic code
To ease the programming of the following exercises some generic code is provided in a library called [`ufun.py`](https://www.dropbox.com/s/1btqa2v71zomyii/ufun.py?dl=0).

### Controlling the LED and generating random numbers
The library provides the code to generate random numbers in a range, and to simplify the control of the LED. Take a look at the code below to understand how it works. We will use it in the following examples.

RED = 0xFF0000
YELLOW = 0xFFFF33
GREEN = 0x007F00
OFF = 0x000000

def random_in_range(l=0,h=1000):
    r1 = ucrypto.getrandbits(32)
    r2 = ((r1[0]<<24)+(r1[1]<<16)+(r1[2]<<8)+r1[3])/4294967295.0
    return math.floor(r2*h+l)

def set_led_to(color=GREEN):
    pycom.heartbeat(False) # Disable the heartbeat LED
    pycom.rgbled(color)

def flash_led_to(color=GREEN, t1=1):
    set_led_to(color)
    time.sleep(t1)
    set_led_to(OFF)

### Connecting your LoPy to WiFi
The LoPy needs to be connected through WiFi to access the local and remote brokers. The code of `connect_to_wifi()` below provides this service and is general enough to work in most contexts.
By properly passing the values `wifi_ssid` and `wifi_passwd` it will try three times to connect to the specified AP, exiting if the operation is not possible. Take a look at the code below to understand how it works. We will use it in all the following examples.

def connect_to_wifi(wifi_ssid, wifi_passwd):
	wlan = WLAN(mode=WLAN.STA)

	for ltry in range(3):
	    print("Connecting to: "+wifi_ssid+". Try "+str(ltry))
	    nets = wlan.scan()
	    for net in nets:
	        if net.ssid == wifi_ssid:
	            print('Network '+wifi_ssid+' found!')
	            wlan.connect(net.ssid, auth=(net.sec, wifi_passwd), timeout=5000)
	            while not wlan.isconnected():
	                machine.idle() # save power while waiting
	            print('WLAN connection succeeded!')
	            flash_led_to(GREEN, 1)
	            print (wlan.ifconfig())
	            break
	    if wlan.isconnected():
	        break
	    else:
	        print('Cannot find network '+wifi_ssid)
	        flash_led_to(RED, 1)
	
	if not wlan.isconnected():
	    print('Cannot connect to network '+wifi_ssid+'. Quitting!!!')
	    sys.exit(1)

## Installing the MQTT client library in the LoPy

The LoPy devices require a MQTT library to write the client application. The code can be found here: https://github.com/pycom/pycom-libraries/tree/master/lib/mqtt

You basically need to download the `mqtt.py` file and copy it in the `/flash/lib` directory of the device. 

## A simple subscriber

The code below represent a simple subscriber. As a first step it connects to the WiFi network available in the lab (remember to properly configure the values to connect to your Wi-Fi access point.)
In this case we will use the broker `test.mosquitto.org` but you can use the one on the Raspberry Pi.

from mqtt import MQTTClient
import time
import pycom

import ufun

wifi_ssid = "LOCAL_AP"
wifi_passwd = ''
broker_addr = "test.mosquitto.org"

def settimeout(duration):
   pass

def on_message(topic, msg):

    print("topic is: "+str(topic))
    print("msg is: "+str(msg))

### if __name__ == "__main__":

ufun.connect_to_wifi(wifi_ssid, wifi_passwd)

client = MQTTClient("dev_id", broker_addr, 1883)
client.set_callback(on_message)

if not client.connect():
    print ("Connected to broker: "+broker_addr)
client.subscribe('lopy/lights')

print('Waiting messages...')
while 1:
    client.wait_msg()

Now, in a terminal on your Raspberry Pi, and using `mosquitto_pub`, write the proper command to send some message to the LoPy.

Analyze now the structure of the MQTT related code. Can you spot the differences with the `Paho` code we saw in class?

## A simple publisher

Let's produce some random data. Copy the code below in your LoPy:

from mqtt import MQTTClient
import time
import pycom

import ufun

wifi_ssid = "LOCAL_AP"
wifi_passwd = ''
#broker_addr = "10.1.1.100"
broker_addr = "test.mosquitto.org"
MYDEVID = "PMdev"

def settimeout(duration):
   pass

def get_data_from_sensor(sensor_id="RAND"):
    if sensor_id == "RAND":
        return ufun.random_in_range()

### if __name__ == "__main__":

ufun.connect_to_wifi(wifi_ssid, wifi_passwd)

client = MQTTClient(MYDEVID, broker_addr, 1883)

if not client.connect():
    print ("Connected to broker: "+broker_addr)

print('Sending messages...')
while True:
    # creating the data
    the_data = get_data_from_sensor()
    # publishing the data
    client.publish(MYDEVID+'/value', str(the_data))
    time.sleep(1)

Now, in a terminal on your Raspberry Pi, and using `mosquitto_sub`, write the proper command to read the generated data.

Analyze now the structure of the MQTT related code. Can you see the differences with the `Paho` code we saw in class?

---

# Final exercises
Now let's work on some final exercises to put together most of what we saw in this lab session. Since you'll have to write some MQTT program for the Raspberry Pi, you have to install the Paho library that I described in class; it's just one step. In the Raspberry Pi execute:

sudo pip install paho-mqtt

## The first
Let's control remotely the color of the LoPy's LED using MQTT.


* block “p1”:
* Each LoPy will be connected to its own local broker in the Raspberry Pi. The LoPy should change the color of its LED according to the "instructions" it receives using MQTT.
* Take advantage of the functions in library `ufun.py` to control the LED.

* block “p2”:
* Contains the program, to be executed on the Rapsberry Pi, that reads 2 parameters: the broker address and the LED color you want that specific LoPy to show. Try to control the LoPy of another group.

## The second
Now repeat the previous exercise but using a unique ("common") broker for the whole lab. It could either be one running in a Raspberry Pi in the lab or remote (e.g., test.mosquitto.org). How will you identify a specific LoPy now?

## The third

For this exercise again you will be using a unique ("common") broker for the whole lab. It could either be one running in a Raspberry Pi in the lab or remote (e.g., test.mosquitto.org).

* block “p1”:
* Reads the values of a specific sensor in the PySense and publish it periodically to the "common" broker
* To get data from the PySense's sensors, you can extend the code of the function `get_data_from_sensor` we used before in the simple publisher example, and use the library [`pysense_s`](https://www.dropbox.com/s/bri65w828jvlms9/pysense_s.py?dl=0).
> WARNING: if the PySense is not available, use the function `get_data_from_sensor` as-is.

* block “p2”:
* Contains the program, to be executed on the Rapsberry Pi, that subscribes to all the values published in the "common" broker by the various groups in the lab, compute the average and visualize the value
* remind that code "p2" will use the Paho library version of MQTT, while the code "p1" will have to use the micropython version.
* you'll have to agree beforehand on the topic values

## The fourth
Modify the previous example so that your LoPy publish the temperature values it gets to your channel in ThingSpeak.