MQTT

Building a MQTT broker using a RPi, sending data via WiFi, actuation using MQTT

Basically this lab is organized so that you can have an hands-on experience with MQTT and learn how to "publish" and "subscribe" to data. To this end you will use:

1. a "sandbox" broker
2. the ThingSpeak broker
3. your own broker

Among other things you will:

1. produce data using the command line
2. produce data by code using the LoPy devices and the Raspberry Pi
3. produce data to your own channel in ThingSpeak

All the code that you will be using is available here.

Hardware

Each group will use:

• A Raspberry Pi 3 with Ubuntu MATE 16.04.2
• A LoPy connected through a PySense board

The various elements are supposed to be connected as indicated in the figure below.

For the next steps you will use one or more terminals (e.g., xterm) connected to your Raspberry Pi via ssh -X.

Installing the MQTT broker

There are various MQTT brokers that can be used with a Raspberry Pi device. A quite complete list can be found here https://github.com/mqtt/mqtt.github.io/wiki/servers

The most widely used are:

• http://mosquitto.org/
• http://www.hivemq.com/
• https://www.rabbitmq.com/mqtt.html
• http://activemq.apache.org/mqtt.html
• https://github.com/mcollina/mosca

For our experiments we will use Eclipse Mosquitto. For more documentation check here: https://mosquitto.org/

Installation steps:

To install the broker and some utilities you need to execute the following steps:

sudo apt-add-repository ppa:mosquitto-dev/mosquitto-ppa
sudo apt-get update
sudo apt-get install mosquitto mosquitto-clients

Once installed, please take a look at its man page.

Managing the broker

To check if the broker is running you can use the command:

sudo netstat -tanlp | grep 1883

note: "-tanlp" stands for: tcp, all, numeric, listening, program

alternatively use:

ps -ef | grep mosquitto

To start and stop its execution use:

sudo /etc/init.d/mosquitto start/stop

if necessary, to avoid that it restarts automatically, do: sudo stop mosquitto

To run the broker execute:

sudo mosquitto –v

note: "-v" stands for "verbose mode" and can be useful at the beginning to see what is going on in the broker. Can be convenient to use a dedicated terminal for the broker to execute in, if the "-v" option is used.

Clients for testing

The broker comes with a couple of useful commands to quickly publish and subscribe to some topic. Their basic syntax is the following.

mosquitto_sub -h HOSTNAME -t TOPIC
mosquitto_pub -h HOSTNAME -t TOPIC -m MSG

More information can be found:

• https://mosquitto.org/man/mosquitto_sub-1.html
• https://mosquitto.org/man/mosquitto_pub-1.html

Some example:

Activate the broker with mosquitto -v in a terminal and open 2 more terminals, more or less like this:

This way you'll control what your broker is doing.

Let's start with a easy one. In one terminal write:

mosquitto_sub -t i/LOVE/Python

the broker terminal should show something like:

the broker registered the subscription request of the new client. Now in another terminal, execute:

mosquitto_pub -t i/LOVE/Python -m "Very well!"

in the broker terminal, after the new registration messages, you'll also see something like:

meaning that the broker received the published message and that it forwarded it to the subscribed client. In the terminal where mosquitto_sub is executing you'll see the actual message appear.

Try now:

mosquitto_pub -t i/love/python -m "Not so well!"

What happened? Are topics case-sensitive?

Another useful option of mosquitto_pub is the following:

mosquitto_pub -t i/LOVE/Python -l

it sends messages read from stdin, splitting separate lines into separate messages. Note that blank lines won't be sent. Give it a try … you basically obtained a MQTT based unidirectional chat channel

qos (Quality of Service). Adding the -q option, for example to the mosquitto_pub you'll see the extra message that are now interchanged with the broker. For example, doing:

mosquitto_pub -t i/LOVE/Python -q 2 -m testing

you'll get:

compare this sequence of messages with the one obtanined with -q 0 or with -q 1.

Retained messages:

Normally if a publisher publishes a message to a topic, and no one is subscribed to that topic the message is simply discarded by the broker. If you want your broker to remember the last published message, you'll have to use the retain option. Only one message is retained per topic. The next message published on that topic replaces the retained message for that topic.

To set the retain message flag you have to add -r using the Mosquitto clients.

So try the following cases, but remember now to start the subscriber after the publisher:

1. Publish a message with the retain message flag not set, like we did before. What happens?
2. Publish a message with the retain message flag set (-r). What happens?
3. Publish several (different) messages with the retain message flag set before starting the subscriber. What happens?
4. Publish a message with the retain message flag not set again. What happens?

Finaly, how do I remove or delete a retained message? You have to publish a blank message with the retain flag set to true which clears the retained message. Try it.

Public brokers

There are also various public brokers in Internet, also called sandboxes. For example:

• iot.eclipse.org
  • more infos at: https://iot.eclipse.org/getting-started#sandboxes
• test.mosquitto.org
  • more infos at: http://test.mosquitto.org/
• broker.hivemq.com
  • more infos at: http://www.hivemq.com/try-out/
    • http://www.mqtt-dashboard.com/

we will always access them through port 1883. Repeat some of the exercise above with one of these sandboxes (remember to use the -h option). Any difference?

Accessing ThingSpeak via MQTT

ThingSpeak is an IoT analytics platform service that allows you to aggregate, visualize and analyze live data streams in the cloud. ThingSpeak provides instant visualizations of data posted by your devices to ThingSpeak. With the ability to execute MATLAB® code in ThingSpeak you can perform online analysis and processing of the data as it comes in. Some of the key capabilities of ThingSpeak include the ability to:

You will see much more about this platform in a later lab session.

Creating a channel

You first have to sign in. Go to https://thingspeak.com/users/sign_up and create your own account. Then you can create your first channel. Like for example:

In the "Private View" section you can get an overview of your data:

Take a look to the other sections. To connect to your channel, you need the data in the API Keys section. In my case it says:

Now, ThingSpeak offers either a REST and a MQTT API to work with channels. See here: https://es.mathworks.com/help/thingspeak/channels-and-charts-api.html
Unfortunately, for the moment, ThingSpeak supports only publishing to channels using MQTT.

Exercise

Anyway, let's publish to your channel field feed some random value using mosquitto_pub.

Consider that:

1. the hostname of the ThinSpeak MQTT service is "mqtt.thingspeak.com"
2. the topic you have to use is channels/<channelID>/publish/fields/field<fieldnumber>/<apikey> where you have to replace:
   • <channelID> with the channel ID,
   • <fieldnumber> with field number that you want to update, and
   • <apikey> with the write API key of the channel.
3. finally, remember that ThingSpeak requires you to:
   • set the PUBLISH messages to a QoS value of 0.
   • set the connection RETAIN flag to 0 (False).
   • set the connection CleanSession flag to 1 (True).
4. more infos here
   https://es.mathworks.com/help/thingspeak/publishtoachannelfieldfeed.html

MQTT clients with MicroPython and the LoPy

First, some basic code

To ease the programming of the following exercises some generic code is provided in a library called ufun.py.

Controlling the LED and generating random numbers

The library provides the code to generate random numbers in a range, and to simplify the control of the LED. Take a look at the code below to understand how it works. We will use it in the following examples.

RED = 0xFF0000
YELLOW = 0xFFFF33
GREEN = 0x007F00
OFF = 0x000000

def random_in_range(l=0,h=1000):
    r1 = ucrypto.getrandbits(32)
    r2 = ((r1[0]<<24)+(r1[1]<<16)+(r1[2]<<8)+r1[3])/4294967295.0
    return math.floor(r2*h+l)

def set_led_to(color=GREEN):
    pycom.heartbeat(False) # Disable the heartbeat LED
    pycom.rgbled(color)

def flash_led_to(color=GREEN, t1=1):
    set_led_to(color)
    time.sleep(t1)
    set_led_to(OFF)

Connecting your LoPy to WiFi

The LoPy needs to be connected through WiFi to access the local and remote brokers. The code of connect_to_wifi() below provides this service and is general enough to work in most contexts.
By properly passing the values wifi_ssid and wifi_passwd it will try three times to connect to the specified AP, exiting if the operation is not possible. Take a look at the code below to understand how it works. We will use it in all the following examples.

def connect_to_wifi(wifi_ssid, wifi_passwd):
	wlan = WLAN(mode=WLAN.STA)

	for ltry in range(3):
	    print("Connecting to: "+wifi_ssid+". Try "+str(ltry))
	    nets = wlan.scan()
	    for net in nets:
	        if net.ssid == wifi_ssid:
	            print('Network '+wifi_ssid+' found!')
	            wlan.connect(net.ssid, auth=(net.sec, wifi_passwd), timeout=5000)
	            while not wlan.isconnected():
	                machine.idle() # save power while waiting
	            print('WLAN connection succeeded!')
	            flash_led_to(GREEN, 1)
	            print (wlan.ifconfig())
	            break
	    if wlan.isconnected():
	        break
	    else:
	        print('Cannot find network '+wifi_ssid)
	        flash_led_to(RED, 1)
	
	if not wlan.isconnected():
	    print('Cannot connect to network '+wifi_ssid+'. Quitting!!!')
	    sys.exit(1)

Installing the MQTT client library in the LoPy

The LoPy devices require a MQTT library to write the client application. The code can be found here: https://github.com/pycom/pycom-libraries/tree/master/lib/mqtt

You basically need to download the mqtt.py file and copy it in the /flash/lib directory of the device.

A simple subscriber

The code below represent a simple subscriber. As a first step it connects to the WiFi network available in the lab (remember to properly configure the values to connect to your Wi-Fi access point.)
In this case we will use the broker test.mosquitto.org but you can use the one on the Raspberry Pi.

from mqtt import MQTTClient
import time
import pycom

import ufun

wifi_ssid = "LOCAL_AP"
wifi_passwd = ''
broker_addr = "test.mosquitto.org"

def settimeout(duration):
   pass

def on_message(topic, msg):

    print("topic is: "+str(topic))
    print("msg is: "+str(msg))

### if __name__ == "__main__":

ufun.connect_to_wifi(wifi_ssid, wifi_passwd)

client = MQTTClient("dev_id", broker_addr, 1883)
client.set_callback(on_message)

if not client.connect():
    print ("Connected to broker: "+broker_addr)
client.subscribe('lopy/lights')

print('Waiting messages...')
while 1:
    client.wait_msg()

Now, in a terminal on your Raspberry Pi, and using mosquitto_pub, write the proper command to send some message to the LoPy.

Analyze now the structure of the MQTT related code. Can you spot the differences with the Paho code we saw in class?

A simple publisher

Let's produce some random data. Copy the code below in your LoPy:

from mqtt import MQTTClient
import time
import pycom

import ufun

wifi_ssid = "LOCAL_AP"
wifi_passwd = ''
#broker_addr = "10.1.1.100"
broker_addr = "test.mosquitto.org"
MYDEVID = "PMdev"

def settimeout(duration):
   pass

def get_data_from_sensor(sensor_id="RAND"):
    if sensor_id == "RAND":
        return ufun.random_in_range()

### if __name__ == "__main__":

ufun.connect_to_wifi(wifi_ssid, wifi_passwd)

client = MQTTClient(MYDEVID, broker_addr, 1883)

if not client.connect():
    print ("Connected to broker: "+broker_addr)

print('Sending messages...')
while True:
    # creating the data
    the_data = get_data_from_sensor()
    # publishing the data
    client.publish(MYDEVID+'/value', str(the_data))
    time.sleep(1)

Now, in a terminal on your Raspberry Pi, and using mosquitto_sub, write the proper command to read the generated data.

Analyze now the structure of the MQTT related code. Can you see the differences with the Paho code we saw in class?

---

Final exercises

Now let's work on some final exercises to put together most of what we saw in this lab session. Since you'll have to write some MQTT program for the Raspberry Pi, you have to install the Paho library that I described in class; it's just one step. In the Raspberry Pi execute:

sudo pip install paho-mqtt

The first

Let's control remotely the color of the LoPy's LED using MQTT.

• block “p1”:

  • Each LoPy will be connected to its own local broker in the Raspberry Pi. The LoPy should change the color of its LED according to the "instructions" it receives using MQTT.
  • Take advantage of the functions in library ufun.py to control the LED.

• block “p2”:

  • Contains the program, to be executed on the Rapsberry Pi, that reads 2 parameters: the broker address and the LED color you want that specific LoPy to show. Try to control the LoPy of another group.

The second

Now repeat the previous exercise but using a unique ("common") broker for the whole lab. It could either be one running in a Raspberry Pi in the lab or remote (e.g., test.mosquitto.org). How will you identify a specific LoPy now?

The third

For this exercise again you will be using a unique ("common") broker for the whole lab. It could either be one running in a Raspberry Pi in the lab or remote (e.g., test.mosquitto.org).

• block “p1”:
  • Reads the values of a specific sensor in the PySense and publish it periodically to the "common" broker
  • To get data from the PySense's sensors, you can extend the code of the function get_data_from_sensor we used before in the simple publisher example, and use the library pysense_s.

WARNING: if the PySense is not available, use the function get_data_from_sensor as-is.

• block “p2”:
  • Contains the program, to be executed on the Rapsberry Pi, that subscribes to all the values published in the "common" broker by the various groups in the lab, compute the average and visualize the value
  • remind that code "p2" will use the Paho library version of MQTT, while the code "p1" will have to use the micropython version.
  • you'll have to agree beforehand on the topic values

The fourth

Modify the previous example so that your LoPy publish the temperature values it gets to your channel in ThingSpeak.