# Part 3: Installing and Using MQTT Broker (Mosquitto) & Node-Red
###### tags: `TA Stuff RP2` `Raspberry Pi Pico`

:::info
This is an extra topic, meaning it is **NOT mandatory** in any way. It is mentioned only because it might be a good solution that fits your project if you decide to self-host your MQTT broker and want to learn more about Node-Red. The following installation scripts are from **CentOS Stream 9** and if you use other Linux distros then the steps and scripts must be slightly different from what you read here.
:::

### Before starting
This tutorial should provide you with some help in installing **Mosquitto** which is an open-source self-hosted message broker that implements the MQTT protocol.

It also provides you with some help installing and using **Node-Red** which is a flow-based development tool for visual programming based on Nodejs and runs on the browser. You can use it to integrate a multitude of different devices and services as well as create your functions. There are also many so-called palettes which are groups of nodes that you can download and install to get more nodes that serve a specific function.

As always, we'll continuously update this walkthrough. **Is there anything that needs to be clarified, or have you experienced some issues? Please add a comment.** You do this by highlighting the text and then you can write a comment on the highlighted part. You need to log in/create an account on HackMD first.

## Installing and Testing Mosquitto
The MQTT protocol provides a lightweight method of carrying out messaging using a publish/subscribe model. This makes it suitable for Internet of Things messaging such as with low-power sensors or mobile devices such as phones, embedded computers, or microcontrollers. The following terms' descriptions were added from [**this link**](https://www.eginnovations.com/documentation/Mosquitto-MQTT/What-is-Mosquitto-MQTT.htm).

In MQTT there are a few basic concepts that you need to understand:
+ **Publish/Subscribe:** In a publish and subscribe system, a device can publish a message on a topic, or it can be subscribed to a particular topic to receive messages.
+ **Messages:** Messages are the information that you want to exchange between your devices. This can either be a command or data.
+ **Topics:** Topics are the way you register interest in incoming messages or how you specify where you want to publish the message.
+ **Broker:** The broker is primarily responsible for receiving all messages, filtering the messages, deciding who is interested in them, and then publishing the message to all subscribed clients.

:::info
In MQTT, a publisher (device/client) publishes messages on a topic and a subscriber must subscribe to that topic to view/read/get the message.
:::

### Basic Installation
**Step 1:** Update the CentOS repository and install the Mosquitto

$ sudo dnf -y install epel-release		# install EPEL repository
$ sudo dnf -y install mosquitto		# Install Mosquitto
$ sudo systemctl start mosquitto              # Starting mosquitto
$ sudo systemctl enable mosquitto             # Make it run on system restart
$ sudo mosquitto -h				# Checking mosquitto vesion and supported protocol versions

**Step 2:**
Open two terminal windows to the Linux, the first one subscribe (subscriber) to the broker and the second one publish (publisher) a message so the first one should receive the message posted by the publisher.

# SUB
$ mosquitto_sub -h localhost -t test
# PUB
$ mosquitto_pub -h localhost -t test -m "hello world"

### Advanced Configuration and Authentication
In the previous steps, the MQTT broker does not need any user authentication, publisher and subscriber access control to specific topic format, and client authorization of using the broker. Following steps we demonstrate how to configure the broker and the last step shows how to test if everything works.

**Step 1:** Set the Mosquitto configuration file. You should add the following lines at the end of the `/etc/mosquitto/mosquitto.conf` file. The comments of what each line does start with `#` and you can read more about it [**here**](https://mosquitto.org/man/mosquitto-conf-5.html).

# It sounds good to activate client expiration to allow brokers to remove client sessions..
persistent_client_expiration 15d

# The messages are firstly stored in memory then saved on disk after the autosave_interval second. The default value is 30 minutes (1800s) and it create a risk of large loss. A reduction to 1 minute sound safer.
autosave_interval 60
persistence true
# The filename to use for the persistent database.
persistence_file mosquitto.db
# The path to persistent database file.
persistence_location /var/lib/mosquitto/
# Set max-keepalive if you get "MQTTException: 2"
max_keepalive 0


# All client must provide id when they pub or sub and the pttern starts "id-" 
clientid_prefixes id-
# Disable unauthenticated access to the broker
allow_anonymous false
# The path to the usernames and passwords file
password_file /etc/mosquitto/passwd
# Give authorization access to specific topic to each user
acl_file /etc/mosquitto/aclfile

**Step 2:** Create a directory for persistence DB if not yet existing and give the ownership to mosquitto.

$ sudo mkdir /var/lib/mosquitto
$ sudo chown mosquitto:mosquitto /var/lib/mosquitto/

**Step 3:** Create users and assign a password to each user with the following commands. The first line creates a password for the admin and the second line creates a user named "`myUser`" and then pops up a message to set a password for this user.

$ sudo mosquitto_passwd  -c /etc/mosquitto/passwd admin
$ sudo mosquitto_passwd  /etc/mosquitto/passwd myUser

**Step 4:** You now need to give the password file ownership to mosquitto so it could access it.

$ sudo chown mosquitto:mosquitto /etc/mosquitto/passwd

**Step 5:** We should give each user read/write access to a specific topic so users can not tamper with each other's topics. We should edit/create an ACL file in this path `/etc/mosquitto/aclfile` and copy the following lines in it. You can see the user `admin` can read system messages and read/write to all topics under `devices/#` and user `myUser` could only read/write to `devices/#`.

user admin
topic read $SYS/#
topic readwrite devices/#

user myUser
topic readwrite devices/#

**Step 6:** Now restart the mosquitto to affect changes.

$ sudo systemctl restart mosquitto

**Step 7:** Then again open two terminal windows and try to subscribe/publish with new settings to see if everything works.
*Subscriber Terminal*

$ mosquitto_sub -h localhost -t 'devices/temp' -q 2 -c -i id-0000 -u myUser -P 'The_myUser_Password'

*Publisher Terminal*

$ mosquitto_pub -h localhost -t 'devices/temp' -m "{'temperature': 18}" -q 2 -i id-0001 -u myUser -P 'The_myUser_Password'
$ mosquitto_pub -h localhost -t 'devices/temp' -m "{'temperature': 20}" -q 2 -i id-0001 -u myUser -P 'The_myUser_Password'
$ mosquitto_pub -h localhost -t 'devices/temp' -m "{'temperature': 25}" -q 2 -i id-0001 -u myUser -P 'The_myUser_Password'

:::success
If all settings are correct then you should see three messages sent from the publisher terminal in the subscriber terminal. You can watch more about MQTT brokers in [**HiveMQ channel here**](https://youtube.com/playlist?list=PLRkdoPznE1EMXLW6XoYLGd4uUaB6wB0wd).
:::
:::info
We set up Node-Red now to receive the MQTT messages in it.
:::
## Installing Node-Red

Node-Red has a great guide on how to install it on different operating systems. There is also a docker container if you want to go down that path. So follow the steps in this [tutorial](https://nodered.org/docs/getting-started/local) to install Node-Red and then you might also want to check the [Securing Node-RED](https://nodered.org/docs/user-guide/runtime/securing-node-red) guide if you are concerned about security. You do not necessarily have to enable HTTPS (if you are on your home network) but it is recommended to enable it later on. It is a good idea however to configure the login credentials.

### Using Node-Red

Using Node-Red should be straightforward. You are supplied with some nodes that perform different functions right from the beginning but you can also install more nodes that provide different integreations such as writing to an [Influxdb](https://www.influxdata.com/) database or sending a message on [Telegram](https://telegram.org/).

Node-Red also provides a great [user guide](https://nodered.org/docs/user-guide/) that goes through what node-red is capable of, and they also have a [concepts](https://nodered.org/docs/user-guide/concepts) page that explains what each concept means.

### Core nodes
Looking at the [Core Nodes](https://nodered.org/docs/user-guide/nodes) page you can see that the "inject" node has no connection box on the left side, that is because it does not take any input and the flow will be going from left to right. It does however have an output connection that will connect to the next node which will have data injected into from the "inject" node.

The opposite applies to the "debug" node. It has an input connection but no output. The debug info will be presented in the debug window on the right sidebar.


Although the "inject" and "debug" nodes are input/output only respectively, it does not mean that you are restricted to only one input and one output. You can have multiple inputs and outputs which allows you to create very complex flows with many different debug points that do not stop the flow itself.

In the picture below you can see that we have 2 inputs and 3 outputs with 2 different debug nodes that will be debugging 2 different members of the msg object. The flow will be output to the debug node, MQTT as well as UDP.


So you can for example have a flow that:
1. Takes input from MQTT
2. Does some formatting/parsing/processing
3. Stores the data in a database
4. Checks if a certain condition applies
5. Sends an alarm message if the condition applies or maybe sends a new MQTT message to an appliance to turn it on.

Some nodes allow for easy control of the amount of messages sent down the flow. This is very useful in case you decide to send messages to your smartphone since you will be avoiding spam. It can also be used to restrict the amount of data stored in the database.

### The function node

The function node is a core node that is super useful and powerful because it allows you to write your functions that manipulate the msg object that holds the data that runs in the flow. This means that you can read/format/parse/modify the data or payload before it goes to the next node. Maybe you are sending temperature and humidity data on MQTT as a string in the format ("30.5,43.7") then you can use the function node to parse the data and convert it to two floats and then store it in the database as two different floats.

The programming language used in the function node is javascript which means that you have access to a good collection of functions that help you parse and format data. It is also a loosely typed language which means that you do not have to specify the type of the data you are storing just like in Python. In difficulty, both Python and JavaScript are considered to be easier to learn for beginners.

Check the [W3 javascript page](https://www.w3schools.com/js/DEFAULT.asp) to learn more about the language.

### Deploying

Once the flow is done you can click the deploy button to enable the flow. However, it is always a good idea to not have any data written immediately to the database. Instead, use a debug node to confirm that the data is in the correct format and is coming in at the correct rate. Also, you should use a limiting node even if your data rate is restricted before coming into Node-Red since it will provide an extra layer of security to prevent flooding the database with too much junk data.

Any changes made to the flow will need to be redeployed. If you try to close the browser Node-Red will warn you for unsaved changes if you have not deployed your latest flow.

Beware however that changing the position of the nodes without changing the actual connections is also considered a modification that prompts a save so you do not always have to save if you have not done any actual modification.

### Installing palettes

Although Node-Red has a good collection of very useful core nodes they will not be enough for most of your project. Luckily we can use the palette manager to install pallets that will give us more nodes that provide different functions and integrations such as MQTT in and out nodes or even HTTP ones. We also can install InfluxDB pallets to be able to store data in an influxDB database.

It is always a good idea to check for palettes since it is very likely that a palette already exists that will make what you are doing easier.

To install a new pallet we go to the palette manager by clicking on the hamburger menu on the right top corner of the screen near the deploy button. From there you should click on the "Manage palette" option.


From there you will get a window that will list all the currently installed palettes. You then can click on the "install" tab to search for a palette to download. We can type "influxdb" in the search to get a search result from which we pick the first one and click on the install button.


After we install a palette we will get the nodes that this palette provides. We can then drag and drop them into our workflow just like any other Node-Red node. You might still want to check the webpage of the palette to get an idea of how to use the nodes provided.

### Node-Red Summary

All in all, Node-Red should be easy to install and relatively easy to get used to. You will need to learn a little javascript but it will not be that big of a problem since you most likely are not going to need to write logic code, only data manipulation since the logic part is done through the different nodes you have at hand. You also have access to more nodes online that you can download which makes life easier.

Hope you give it a try if you think that it is something that adds value to your project and if so make sure to let us know how it went
🤩

## Connecting Node-Red to Mosquitto MQTT Broker
In this section, you can see steps to publish a message from **Raspberry Pi Pico W** to an MQTT topic on Mosquitto subscribe to the same topic from Node-Red, and see the message in a debug node.

**Step 1:** Add an `"mqtt in"` and a `"Debug"` node to your Node-Red.
**Step 2:** Connect the `"mqtt in"` output to the `"Debug"` input.
**Step 3:** Double click on `"mqtt in"` node and then click on the edit button in front of *server* to add a new server. Then, follow the numbered steps to add all required information and setup a `"Subscriber"` to the Mosquitto.

</br>


</br>


**Step 4:** To apply changes you must click on `"Deploy"` button on top right corder.

**Step 5:** If all required information is correct you should see `"connected"` under `"mqtt in"` node. If not then there is mismatch between Mosquitto and information provided here.

**Step 6:** Download [**this project**](https://github.com/iot-lnu/pico-w/tree/main/network-examples/N5_WiFi_Mosquitto_Node-Red_Test) from LNU GitHub and change all required information in the `config.py` file and upload all libraries and `main.py` to your Raspberry Pi Pico and reset the board. The code publishes a value named `indoorTemp` by creating a random number between 18 and 35 and sending it to Mosquitto. You should see the incoming messages in the `Debug` panel on the right side of Node-Red.

<br/>

:::info
There are different ways to visualize the received values, one could be to install [**Node-Red-Dashboard here**](https://flows.nodered.org/node/node-red-dashboard) and use the widgets to illustrate your data.
:::

<style>

html, body, .ui-content {
background-color: #333;
color: #ddd;
}

body > .ui-infobar {
display: none;
}

.ui-view-area > .ui-infobar {
display: block;
}

.markdown-body h1,
.markdown-body h2,
.markdown-body h3,
.markdown-body h4,
.markdown-body h5,
.markdown-body h6 {
color: #ddd;
}

.markdown-body h1,
.markdown-body h2 {
border-bottom-color: #ffffff69;
}

.markdown-body h1 .octicon-link,
.markdown-body h2 .octicon-link,
.markdown-body h3 .octicon-link,
.markdown-body h4 .octicon-link,
.markdown-body h5 .octicon-link,
.markdown-body h6 .octicon-link {
color: #fff;
}

.markdown-body img {
background-color: transparent;
}

.ui-toc-dropdown .nav>.active:focus>a, .ui-toc-dropdown .nav>.active:hover>a, .ui-toc-dropdown .nav>.active>a {
color: white;
border-left: 2px solid white;
}

.expand-toggle:hover,
.expand-toggle:focus,
.back-to-top:hover,
.back-to-top:focus,
.go-to-bottom:hover,
.go-to-bottom:focus {
color: white;
}

.ui-toc-dropdown {
background-color: #333;
}

.ui-toc-label.btn {
background-color: #191919;
color: white;
}

.ui-toc-dropdown .nav>li>a:focus,
.ui-toc-dropdown .nav>li>a:hover {
color: white;
border-left: 1px solid white;
}

.markdown-body blockquote {
color: #fff;
}

.markdown-body table tr {
background-color: #5f5f5f;
}

.markdown-body table tr:nth-child(2n) {
background-color: #4f4f4f;
}

.markdown-body code,
.markdown-body tt {
color: #f;
background-color: rgba(230, 230, 230, 0.36);
}

a,
.open-files-container li.selected a {
color: #5EB7E0;
}

.markdown-body pre {
color: black;
}
</style>