## Building a weathersensor and displaying the data through Amazon AWS
Gustav Skog - gs222nz - IDT305 July 2020

The project was made in the course *Applied Internet of things*, a distance course at Linneaus University the summer 2020.

This project goes over how to set up a Pycom weather sensor, how to send the data to Pybytes through Sigfox, from Pybytes to Amazon AWS Iot and how to visualize the data in webpage with Chartjs. Following this tutorial with all the steps should take about 2-3 hours.

Due to the lenght of this tutorial some parts are omitted and instead linked to external guides. I've tried to clarify parts that I found less documented.

The end result is a temperature and humidity sensor, with a corresponding live dashboard that is accessed at http://btp-dashboard.s3-website.eu-central-1.amazonaws.com/.

### Objective
The goal with the project was to try out the available cloud features available for Iot devices.

The product is implemented in my families summer cottage, and can show the continual changes in temperature and humidity over time. There is no wifi available, so the connection is made with Sigfox. It is placed in a newly built outside room with glass walls, and can show how hot and cold it is in relation to the outside temperature.

### Physical material
>
| Thing | Purpose | Link |
|----------|:-------------:|------:|
| LoPy4 with breakout board | Microcontroller that is connected to the sensors | [€56](https://pycom.io/product/lopy4-multipack/) |
| DHT11| Temperature and humidity sensor | [35kr](https://www.styrahem.se/p/382/luftfuktighet-och-temperaturgivare-dht11) |
| Breadboard | For easier development | [€5.25](https://www.ebay.co.uk/itm/MB-102-830-Kontakte-Breadboard-gros-Steckbrett-Raspberry-Pi-Experimentier/252860727462?hash=item3adfac80a6:g:wC4AAOSwn01b3zoN)|
| Jumpercables | To connect the sensor | [€5.35](https://www.ebay.co.uk/itm/65-Jumper-Wire-Steckbrett-Steckboard-Steckbrucken-Raspberry-Pi-Arduino-Kabel/252715059890?hash=item3ad6fdcab2:g:FGgAAOSwXWhb3ZiW) |
| MicroUSB charger | To power the LoPy4 | n/a |

The project is made with the LoPy4 from Pycom. This has built in Sigfox that enables the data to easily be sent, and a good platform Pybytes to receive the data in the cloud. The DHT11 sensor is cheap and give both temperature and humidity and has good support with the available library. Breadboard and wires are optional, but was useful during development. For powering the device I've used an old MicorUSB charger.

### Computer setup
I've used VS Code for writing and the PyMakr plugin for connecting and uploading the code to the LoPy4. To install simply search for Pymakr in VS Code extensions. You also need to install Node.JS. To setup Sigfox the device need to be programmed through Pycom Firmware Update, after creating a Sigfox account and linking it to Pybytes.

## Putting everything together
The DHT11 temperature and humidity sensor is connected to Pin 23 on the LoPy4, as well as 3.3V and Ground pins on the breakout board. The LoPy4 can only handle a maximum voltage of 3.3V at the input pins, but the sensor work well with 3.3V and nothing else is needed. The device is powered though USB on the breakout board.

The breadboard is only used for a development setup, as the jumper wires are fragile. To make a more permament a housing for the LoPy4 and short cables for the sensor would protect the board as well as make it more tidy during continued development.

### Platform
The project uses a few services to send the data to the dashboard. The data is sent through Sigfox to Pybytes, and from Pybytes to Amazon AWS Iot Core, saved to a DynamoDB and presented on a website hosted in a S3 Bucket. Pybytes is free and with the purchase of a LoPy4 1 year of Sigfox is included. After 1 year Sigfox uses a subscription model. All the Amazon services are accessed through the free tier.

>
| Services | Purpose | Price |
|----------|:-------------:|------:|
| Sigfox| Sending data without wifi | Free 1 year |
| Pybytes | Reciving the data in the cloud | Free |
| Amazon AWS Iot | As an MQTT broker integrated with pybytes | Free |
| Amazon AWS DynamoDB | Database to save data in AWS | Free|
| Amazon AWS S3 | Online website hosting | Free |

The implementation uses Pycoms built in function pybytes.send_signal() to send the measured data to the Pybytes platform. Pybytes has four possible integrations for sending the data to cloud services; Amazon AWS, Microsoft Azure, Goolge Iot and Webhooks. At first I tried Microsoft Azure (which gives 1 year free trial to many services) but found it confusing and very business centric, focusing more on large scale solutions way out of my scope.Then I tried Amazon AWS Iot Core, which also is for large scale implementations, but I was able to find more information and examples how to get it working. AWS also offers a free tier, which has worked well for everything I've done.

### The code
The LoPy4 is running a Micropython program that collects temperature and humidity every third minute, and send the average to Pybytes every 30 minutes.

# ===  Temperature and humidity sensor  ===
# ===          Gustav Skog, 2020        ===

from machine import Pin
import pycom
import time
import math
from dht import DHT # https://github.com/JurassicPork/DHT_PyCom

th = DHT(Pin('P23', mode=Pin.OPEN_DRAIN), 0)

print("Starting program")

# Sends the data after 10 samples, ie every 30 minutes. 
sampleSize = 10
timeBetweenSamples = 3*60

temperatureList = []
humidityList = []

def Average(list): 
    return sum(list) / len(list) 

pycom.heartbeat(False)

while True:
    for i in range(0, sampleSize):
        result = th.read()
        while not result.is_valid():
            time.sleep(.5)
            result = th.read()
        print('Temp:', result.temperature)
        print('RH:', result.humidity)
        
        # Add measured values to list. 
        temperatureList.append(result.temperature)
        humidityList.append(result.humidity)
        
        # Wait for some time before next measurement
        time.sleep(timeBetweenSamples) 
    
    # Calculate averages for measured data
    temperatureAverage = Average(temperatureList)
    humidityAverage = Average(humidityList)

    # Round to 1 decimal point
    roundedTemperature = round(temperatureAverage, 1)
    roundedHumidity = round(humidityAverage, 1)
    pybytes.send_signal(1, roundedTemperature)
    pybytes.send_signal(2, roundedHumidity)
    
    print(temperatureAverage)
    
    # Clear the lists
    temperatureList = []
    humidityList = []
    time.sleep(5)

## Transmitting the data / connectivity


The data is sent from LoPy4 to Pybytes platform through the Sigfox network. Sigfox only allows for 144 messages per day, and I chose to send one data message about every 30 minutes.

From Pybytes the sensordata is sent to AWS Iot Core that acts as a MQTT broker.

### Set up Pybytes and AWS Iot Core
To send data from Pybytes, you need to configure an integration. The official documentation show it well, and it's straighforward in the interface. Worth noting is your chosen message topic, here /telemetry/weatherdata. This is the MQTT topic that Pybytes will send to, and your AWS Iot need to listen to.

Pybytes publishes all the messages that it receives from the selected device to the message topic */telemetry/weatherdata* and the messages look like this: ```json = {
"payload" : "24",
"signal": "1"
} ```
where signal is the chosen signal and payload is the value in *pybytes.send_signal(1, 24)*.

You can test the MQTT in Iot Core. Here you can specify a topic to temporalily subscribe to, */telemetry/weatherdata* and also send messages to the topic. This is great while experimenting as you can send data without your Pycom board.

### Saving data in DynamoDB
AWS Iot Core uses *Rules* to react to the incoming MQTT messages. We will specify a Rule to save our sensordata to a DynamoDB database, and also add a timestamp for when the data arrived.

In Iot Core we choose *Rule* and *Add new Rule*. Here we choose a name for the rule, which messages to subscribe to and what will happen when the Rule is triggered.
We write ```= SELECT * FROM "/telemetry/weatherdata" ``` to select everything that is published to */telemetry/weatherdata*. That means; every time a message is published to this topic, the rule is triggered.
Since we want to also add a timestamp, we can use the built in function ```timestamp()``` by writing ```sql= SELECT *, timestamp as timestamp() FROM "/telemetry/weatherdata"```

Next we choose what happens when the rule is triggered, here we choose *Insert message into DynamoDB table*, and since we don't have a table yet we create a new one by pressing *Create new resource*. Here we choose a name for the table, name and datatype for *Primary key* as well as *Sort key*. For *Primary key* we use Signal and for *Sort key* we use Timestamp. Both have *Number* as data type.

Then we can go back and use our newly created table to define the rule. For *Partition key value* we want to use ```${signal}``` and for Timestamp we want to use the timestamp function; ```${timestamp()}```. This will take the signal value from our MQTT message and put in the *Signal* column, and the timestamp in the *Timestamp* column of our table. Under *Write message data to this column* we put Message, this means that the whole Published message will be put in a Message column of the table.


In the end our DynamoDB table will look like this. Note that the timestamp value is written in Epoch (seconds since January 1 1970).

## Presenting the data
The data is presented with chartsjs library on a HTML page. The data is preserved in the DynamoDB, and the frontend fetches the data to display in the dashboard. For making this I found [this guide](https://devan.codes/blog/2016/12/7/building-an-iot-system-using-the-onion-omega-and-amazon-aws) to be very helpful.


<!-- index.html -->
<!DOCTYPE html>
<html lang="en">

<head>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <meta name="description" content="">
    <meta name="author" content="">

    <title>BTp weather monitor</title>

    <!-- Bootstrap core CSS -->
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css">

</head>

<body> 
    <!-- The large heading -->
    <div class="container">
        <br />
        <div class="jumbotron text-center">
            <h1>Temperatur och luftfuktighet</h1>
            <p class="lead">Bodatorp</p>
        </div>

        <div class="row">

            <div class="col-md-6">

                <!-- This is where the temperaturegraph is drawn -->
                <canvas id="temperaturegraph" class="inner cover" width="500" height="320"></canvas>

                <br />
            </div>

            <div class="col-md-6">
                <!-- This is where the humiditygraph is drawn -->
                <canvas id="humiditygraph" class="inner cover" width="500" height="320"></canvas>

                <br />
            </div>
        </div>
    </div> <!-- /container -->
    
    <!-- Externally liniked Javascripts -->
    <script src="https://sdk.amazonaws.com/js/aws-sdk-2.710.0.min.js"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/moment.js/2.22.2/moment.min.js"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/Chart.js/2.9.3/Chart.js"></script>
    <script src="drawgraphs.js"></script>
</body>

</html>

// drawgraphs.js
// === Script for fetching data from AWS and displaying it in line graphs

// AWS Configuration
AWS.config.region = 'eu-central-1'; // Region
AWS.config.credentials = new AWS.Credentials('AKIATH6OJRB4BWGQFB5M', 'w/4MxIUHk3bGF659Uk2BkdI0E3A5qLW/tJeXq8mF');

var dynamodb = new AWS.DynamoDB();
var temperatureParams = {
    TableName: 'weatherSensorData3',
    KeyConditionExpression: '#signal = :signalnumber',
    ExpressionAttributeNames: {
        "#signal": "Signal"
    },
    ExpressionAttributeValues: {
        ":signalnumber": { "N": "1" }
    }
};
var humidityParams = {
    TableName: 'weatherSensorData3',
    KeyConditionExpression: '#signal = :signalnumber',
    ExpressionAttributeNames: {
        "#signal": "Signal"
    },
    ExpressionAttributeValues: {
        ":signalnumber": { "N": "2" }
    }
};

// Run script on document loads
document.onload = onloadFunction();

// Arrays to store temperature values
var temperatureValues = [];
var humidityValues = [];

function onloadFunction() {

    // Fetch temperature data from database
    dynamodb.query(temperatureParams, function (err, data) {
        if (err) {
            console.log(err);
            return null;
        }
        else {
            console.log(data);
            // Parse the data values
            for (var i in data['Items']) {
                var itemValueTemperature = parseFloat(data['Items'][i]['Message']['M']['payload']['N']);
                var itemValueTimestampEpoch = parseInt(data['Items'][i]['Timestamp']['N']);
                var itemValueTimestamp = new Date(itemValueTimestampEpoch)

                // Put data into the local data arrays
                temperatureValues.push({
                    x: itemValueTimestamp,
                    y: itemValueTemperature
                });
            }
            // Make the graph
            makeTemperatureGraph();
        }
    })

    // Fetch humidity data from database
    dynamodb.query(humidityParams, function (err, data) {
        if (err) {
            console.log(err);
            return null;
        }
        else {
            console.log(data);

            // Parse the data values
            for (var i in data['Items']) {
                var itemValueHumidity = parseFloat(data['Items'][i]['Message']['M']['payload']['N']);
                var itemValueTimestampEpoch = parseInt(data['Items'][i]['Timestamp']['N']);
                var itemValueTimestamp = new Date(itemValueTimestampEpoch)

                // Put data into the local data arrays
                humidityValues.push({
                    x: itemValueTimestamp,
                    y: itemValueHumidity
                });
            }
            // Make the graph
            makeHumidityGraph();
        }
    })
}

// Functions for drawing the graphs, with corresponding data
function makeTemperatureGraph() {
    new Chart(document.getElementById("temperaturegraph"), {
        type: 'line',
        data: {
            datasets: [{
                data: temperatureValues,
                label: "Omgivande temperatur",
                borderColor: "#3e95cd",
                fill: false
            }]
        },
        options: {
            title: {
                display: true,
                text: 'Temperaturdata'
            },
            legend: {
                display: false
            },
            scales: {
                xAxes: [{
                    type: 'time',
                }]
            }
        }
    })
}

function makeHumidityGraph() {
    new Chart(document.getElementById("humiditygraph"), {
        type: 'line',
        data: {
            datasets: [{
                data: humidityValues,
                label: "Omgivande luftfuktighet",
                borderColor: "#48c9b0",
                fill: false
            }]
        },
        options: {
            title: {
                display: true,
                text: 'Luftfuktighetsdata'
            },
            legend: {
                display: false
            },
            scales: {
                xAxes: [{
                    type: 'time',
                }]
            }
        }
    })
}

## Finalizing the design
The device is running now and continually sending data, and the dashboard can be accessed here: http://btp-dashboard.s3-website.eu-central-1.amazonaws.com/ It has been a fun project, and long distance transmission with LoRa or Sigfox really opens up Iot implementations in more remote places.

The next steps in the project should be making a more permament setup, with a 3D-printed case. The device could also include other types of data, such as wind speed or mm of rain. To be able to send this additional data, and maybe more frequently, I need to explore the possibility to send all the data in the same payload, as Sigfox only allows for 144 messages.