# 13. TinyML hands-on examples
###### tags: `corsounipd2022`

> Based on:
> * https://github.com/tinyMLx/courseware
> * https://blog.tensorflow.org/2019/11/how-to-get-started-with-machine.html

:::info
The code of this section is [here ](https://www.dropbox.com/sh/u73b82rr9crtzwt/AAALzsmqKDWiGLGu4wqW5eqga?dl=0)
:::

In this lab we will see two practical example using TinyML with two different platforms.
The first lab will work at a lower level using TensorFlow Lite Micro, the second one will use a cloud platform called EdgeImpulse that allows to develop TinyML applications without writing a single line of code.

# Lab 1: TensorFlow Lite Micro

[Arduino](https://www.arduino.cc/) is one of the main actors in making Machine Learning available on simple devices. In this first lab, we will work on how to install and run [TensorFlow Lite Micro](https://www.tensorflow.org/lite/microcontrollers/overview ) examples on a [Arduino Nano 33 BLE Sense](https://store.arduino.cc/nano-33-ble-sense).

This board is one of the most used devices for TinyML experiments. It has an [Arm Cortex-M4](https://developer.arm.com/Processors/Cortex-M4) microcontroller running at 64 MHz with 1MB Flash memory and 256 KB of RAM.

It also has a variety of onboard sensors meaning potential for various Tiny ML applications:


As the name suggests, it has Bluetooth LE connectivity so you can send data (or inference results) to a laptop, mobile app or other BLE boards and peripherals. Connecting the BLE Sense board over USB is an easy way to capture data and add multiple sensors to single board computers without the need for additional wiring or hardware — a nice addition to a Raspberry Pi, for example.

To program this board, you can use the [Arduino Web Editor](https://create.arduino.cc/editor) or install the [Arduino IDE](https://www.arduino.cc/en/main/software).


The inference examples for TensorFlow Lite for Microcontrollers are packaged and available through the Arduino Library manager. For example, the `micro_speech` allows to recognize, using TensorFlow Lite Micro, voice keywords. It has a simple vocabulary of “yes” and “no”.

{%youtube z0pEGtAXN-g %}

<!--
-->

Remember this model is running locally (no Internet connection) on a microcontroller with only 256KB of RAM, so don’t expect commercial ‘voice assistant’ level accuracy.

Regarding power use, the figures below show the evolution when "thinking" and when showing the results with the LED on.




## Training a TensorFlow Lite Micro model for Arduino
:::info
Based on:
https://blog.tensorflow.org/2019/11/how-to-get-started-with-machine.html
:::

This section describes the steps to deploy on a Nano 33 the TinyML code necessary to "understand" some specific movements. The objective is to detail the process involved so that all the necessary elements, from the creation of the dataset to the final deployment on the HW are shown.

:::warning
DISCLAIMER:
This experiment is simply an example on how to use ML to perform a task. It is oriented to **non-experts** on neural networks and the Tensorflow framework.
Training neural network models is a long and difficult process, easily full of frustration...
:::

This example relates to what is known as **predictive maintenance**. The idea is to recognize a pair of movements that are supposed to indicate the regular movement of a machine and alert when a non-standard movement is detected

We’ll capture motion data from the Arduino Nano 33 BLE Sense board, import it into TensorFlow to train a model, and deploy the resulting classifier onto the board.

## Setting up the Arduino IDE

Following the steps below sets up the Arduino IDE application used to both upload inference models to your board and download training data from it in the next section. There are a few more steps involved than using Arduino Create web editor because we will need to download and install the specific board and libraries in the Arduino IDE.

* In the Arduino IDE menu select Tools > Board > Boards Manager…
* Search for “Nano BLE” and press install on the board
* When it’s done close the Boards Manager window


* Now go to the Library Manager Tools > Manage Libraries…
* Search for and install the `Arduino_TensorFlowLite` library
* Next search for and install the `Arduino_LSM9DS1` library:


Finally, plug the micro USB cable into the board and your computer and:
* Choose the board Tools > Board > Arduino Nano 33 BLE

## Streaming sensor data from the Arduino board

First, we need to capture some training data. You can capture sensor data logs from the Arduino board.

We’ll be using a pre-made sketch `IMU_Capture.ino` which does the following:
* Monitor the board’s accelerometer and gyroscope
* Trigger a sample window on detecting significant linear acceleration of the board
* Sample for one second at 119Hz, outputting CSV format data over USB
* Loop back and monitor for the next gesture

:::info
The sensors we choose to read from the board, the sample rate, the trigger threshold, and whether we stream data output as CSV, JSON, binary or some other format are all customizable in the sketch running on the Arduino. There is also scope to perform signal preprocessing and filtering on the device before the data is output to the log. For now, you can just upload the sketch and get to sampling.
:::

## Visualizing live sensor data log from the Arduino board

With that done, we can now visualize the data coming off the board. We’re not capturing data yet — this is just to give you a feel for how the sensor data capture is triggered and how long a sample window is. This will help when it comes to collecting training samples.
* In the Arduino IDE, open the Serial Plotter Tools > Serial Plotter
* If you get an error that the board is not available, reselect the port:
* Tools > Port > portname (Arduino Nano 33 BLE)
* Pick up the board and test different movements possibilities
* You should see a live graph of the sensor data capture:

{%youtube 2K2heDnBCCs %}

## Capturing gesture training data

To capture data as a CSV log to upload to TensorFlow, you can use Arduino IDE > Tools > Serial Monitor to view the data and export it to your desktop machine:
* Reset the board by pressing the small white button on the top
* Pick up the board in one hand (picking it up later will trigger sampling)
* In the Arduino IDE, open the Serial Monitor Tools > Serial Monitor
* Make the movement you would like to be recognized
* Repeat the movement capture step 10 or more times to gather more data
* Copy and paste the data from the Serial Console to a text file
* Clear the console window output and repeat all the steps above, this time with another movement to be recognized

Note the first line of your two csv files should contain the fields aX,aY,aZ,gX,gY,gZ.


:::info
Linux tip: if you prefer you can redirect the sensor log output from the Arduino straight to a .csv file on the command line. With the Serial Plotter / Serial Monitor windows closed use:

$ cat /dev/cu.usbmodem[nnnnn] > sensorlog.csv
:::

:::success
In this example we recorded two movements: `lateral.csv` and `updown.csv`
:::

## Training in TensorFlow

We’re going to use [Google Colab](https://colab.research.google.com/) to train our machine learning model using the data we collected from the Arduino board in the previous section. Colab provides a [Jupyter notebook](https://jupyter.org) that allows us to run our TensorFlow training in a web browser. *A locally running Jupyter notebook could obviously used, too.*

:::info
Let's start first with a [**simple NN example with Tensorflow and Keras**](https://drive.google.com/file/d/1Otw_RTT7mmpfvymIjHrp80TOR4NaBljj/view?usp=sharing).
:::

Ok, let's continue!
The Colab will guide you through the following steps to train the model:

* Setup Python Environment
* Upload the movements data
* Parse and prepare the data
* Build & Train the Model
* Convert the Trained Model to TensorFlow Lite
* Encode the Model in an Arduino Header File

The final step generates the `model.h` file to download and include in our Arduino IDE gesture classifier project in the next section.

[
](https://colab.research.google.com/drive/1QUDPaVpqZIlLiJYwtp8iCXKt72yylVqz)

[Upload the shared file: `arduino_tinyml_seminar.ipynb` ](https://www.dropbox.com/sh/6zhgyfs9znmgeca/AADbIlVtpS50Kg5A2glqG1lka?dl=0)

## Classifying IMU Data

Now, by combining in a project the `model.h` file we just trained and downloaded from Colab in the previous section and the `IMU_Classifier.ino` file, we obtain our movement classifier.

* Upload the sketch: Sketch > Upload
* Open the Serial Monitor: Tools > Serial Monitor
* Perform some gestures
* The confidence of each gesture will be printed to the Serial Monitor (0 = low confidence, 1 = high confidence)


Congratulations, you’ve just trained your first TinyML application for Arduino!

---
---
# Lab 2: An example with Edge Impulse

Edge Impulse is the leading development platform for machine learning on edge devices. It is for developers but also used by enterprises.

[](https://www.edgeimpulse.com)

It allows to cover all the phases in a ML base IoT project, from training to deployment:

[](https://studio.edgeimpulse.com/login)

Currently it can handle the [following hardware:](https://docs.edgeimpulse.com/docs/fully-supported-development-boards)


With different development boards, data can be collected using the [Data forwarder](https://docs.edgeimpulse.com/docs/cli-data-forwarder) or the [Edge Impulse for Linux SDK](https://docs.edgeimpulse.com/docs/edge-impulse-for-linux), and deploy the model back to the device with the [Running your impulse locally tutorials](https://docs.edgeimpulse.com/docs/running-your-impulse-locally-1). **You can also use your [Mobile phone.](https://docs.edgeimpulse.com/docs/using-your-mobile-phone)**

:::info

Click [here :mortar_board: to create an edge impulse account](https://studio.edgeimpulse.com/signup). Many of the CLI tools require the user to log in to connect with the Edge Impulse Studio.
:::

### Edge Impulse CLI Installation

The Edge Impulse CLI is used to control local devices, act as a proxy to synchronise data for devices that don't have an internet connection, and to upload and convert local files. The CLI consists of seven tools, the most important are:

* `edge-impulse-daemon` - configures devices over serial, and acts as a proxy for devices that do not have an IP connection.
* `edge-impulse-uploader` - allows uploading and signing local files.
* `edge-impulse-data-forwarder` - a very easy way to collect data from any device over a serial connection, and forward the data to Edge Impulse.
* `edge-impulse-run-impulse` - show the impulse running on your device.

Installation instructions are available here: https://docs.edgeimpulse.com/docs/cli-installation

:::success
Anyway, recent versions of Google Chrome and Microsoft Edge can connect directly to fully-supported development boards, without the CLI. More later...
:::

### Ingestion service
> https://docs.edgeimpulse.com/reference/ingestion-api

The ingestion service is used to send new device data to Edge Impulse. It's available on both HTTP and HTTPS endpoints, and requires an API key to authenticate. Data needs to be sent in the [Edge Impulse Data Acquisition](https://docs.edgeimpulse.com/docs/data-acquisition-format) format, and is optionally signed with an HMAC key. Data with invalid signatures will still show up in the studio, but will be marked as such, and can be excluded from training.

## Wio Terminal
> https://www.seeedstudio.com/Wio-Terminal-p-4509.html

The Wio Terminal is an ATSAMD51-based microcontroller with both Bluetooth and Wi-Fi connectivity powered by Realtek RTL8720DN.


The SAM D51 micro-controller series is targeted for general purpose applications using the 32-bit ARM® Cortex®-M4 processor with Floating Point Unit (FPU), running up to 120 MHz, up to 1 MB Dual Panel Flash with ECC, and up to 256 KB of SRAM with ECC. The Wio Terminal is integrated with a 2.4” LCD Screen, onboard IMU (LIS3DHTR), microphone, buzzer, microSD card slot, light sensor, infrared emitter (IR 940nm).

It is compatible with Arduino and MicroPython, but currently wireless connectivity is only supported by Arduino.

### Connecting to Edge Impulse
> For more details, [please also see here.](https://wiki.seeedstudio.com/Wio-Terminal-TinyML-EI-1/)

Connect the Wio Terminal to your computer. Entering the bootloader mode by sliding the power switch twice quickly.

An external drive named Arduino should appear in your PC. Drag the the downloaded [Edge Impulse uf2 firmware files](http://files.seeedstudio.com/wiki/Wio-Terminal-Edge-Impulse/res/EdgeImpulse.uf2) to the Arduino drive. Now, Edge Impulse is loaded on the Wio Terminal!

From a command prompt or terminal, run:

edge-impulse-daemon

This will start a wizard which will ask you to log in, and choose an Edge Impulse project. If you want to switch projects run the command with --clean.


That's all! Your device is now connected to Edge Impulse. To verify this, go to your Edge Impulse project, and click Devices. The device will be listed here.


:::info
With Chrome these step can be avoided and use WebUSB:
> see https://www.edgeimpulse.com/blog/collect-sensor-data-straight-from-your-web-browser
:::

## Continuous motion recognition
> https://docs.edgeimpulse.com/docs/continuous-motion-recognition
> https://wiki.seeedstudio.com/Wio-Terminal-TinyML-EI-2/

In this tutorial, you'll use machine learning to build a gesture recognition system that runs on a microcontroller. This is a hard task to solve using rule based programming, as people don't perform gestures in the exact same way every time. But machine learning can handle these variations with ease. You'll learn how to collect high-frequency data from real sensors, use signal processing to clean up data, build a neural network classifier, and how to deploy your model back to a device. At the end of this tutorial you'll have a firm understanding of applying machine learning in embedded devices using Edge Impulse.

### Collecting your first data

With your device connected we can collect some data. In the studio go to the **Data acquisition tab**. This is the place where all your raw data is stored, and - if your device is connected to the remote management API - where you can start sampling new data.

Under **Record new data**, select your device, set the label to `updown`, the sample length to 10000, the sensor to Built-in accelerometer and the frequency to 62.5Hz. This indicates that you want to record data for 10 seconds, and label the recorded data as `updown`. You can later edit these labels if needed.


After you click Start sampling move your device up and down in a continuous motion. In about twelve seconds the device should complete sampling and upload the file back to Edge Impulse. You see a new line appear under 'Collected data' in the studio. When you click it you now see the raw data graphed out. As the accelerometer on the development board has three axes you'll notice three different lines, one for each axis.

You'll get a graph like this:

Machine learning works best with lots of data, so a single sample won't cut it. Now is the time to start building your own dataset. For example, use the following four classes, and record around 3 minutes of data per class:

* Idle - just sitting on your desk while you're working.
* Lateral - moving the device left and right.
* Updown - moving the device up and down.

Up to get to this point:

The tool warn us that we have too few data... but it is an example...

### Designing an Impulse

With the training set in place you can design an **impulse**. An impulse takes the raw data, slices it up in smaller windows, uses signal processing blocks to extract features, and then uses a learning block to classify new data. Signal processing blocks always return the same values for the same input and are used to make raw data easier to process, while learning blocks learn from past experiences.

For this example we'll use the '**Spectral analysis**' signal processing block. This block applies a filter, performs spectral analysis on the signal, and extracts frequency and spectral power data. Then we'll use a 'Neural Network' learning block, that takes these spectral features and learns to distinguish between the three (idle, lateral, updown) classes.

Go to Create impulse, set the window size to 2000 (you can click on the 2000 ms. text to enter an exact value), the window increase to 80, and add the 'Spectral Analysis' and 'Classification (Keras)' blocks. Then click Save impulse.


### Configuring the spectral analysis block

To configure your signal processing block, click **Spectral features** in the menu on the left. This will show you the raw data on top of the screen (you can select other files via the drop down menu), and the results of the signal processing through graphs on the right. For the spectral features block you'll see the following graphs:

* After filter - the signal after applying a low-pass filter. This will remove noise.
* Frequency domain - the frequency at which signal is repeating (e.g. making one wave movement per second will show a peak at 1 Hz).
* Spectral power - the amount of power that went into the signal at each frequency


A good signal processing block will yield similar results for similar data. If you move the sliding window (on the raw data graph) around, the graphs should remain similar. Also, when you switch to another file with the same label, you should see similar graphs, even if the orientation of the device was different.

Once you're happy with the result, click Save parameters. This will send you to the 'Generate features' screen. In here you'll:

* Split all raw data up in windows (based on the window size and the window increase).
* Apply the spectral features block on all these windows.

Click **Generate features** to start the process.


Afterwards the 'Feature explorer' will load. This is a plot of all the extracted features against all the generated windows. You can use this graph to compare your complete data set. E.g. by plotting the height of the first peak on the X-axis against the spectral power between 0.5 Hz and 1 Hz on the Y-axis.

**A good rule of thumb is that if you can visually separate the data on a number of axes, then the machine learning model will be able to do so as well.**


### Configuring the neural network

With all data processed it's time to start training a neural network. Neural networks are a set of algorithms that are designed to recognize patterns. The network that we're training here will take the signal processing data as an input, and try to map this to one of the four classes.

So how does a neural network know what to predict? A neural network consists of layers of neurons, all interconnected, and each connection has a weight. One such neuron in the input layer would be the height of the first peak of the X-axis (from the signal processing block); and one such neuron in the output layer would be wave (one the classes). When defining the neural network all these connections are intialized randomly, and thus the neural network will make random predictions. During training we then take all the raw data, ask the network to make a prediction, and then make tiny alterations to the weights depending on the outcome (this is why labeling raw data is important).

This way, after a lot of iterations, the neural network learns; and will eventually become much better at predicting new data. Let's try this out by clicking on NN Classifier in the menu.

Set **'Number of training cycles' to 1**. This will limit training to a single iteration. And then click Start training.


The figure shows the training performance after a single iteration. On the top a summary of the accuracy of the network, and in the middle a confusion matrix. This matrix shows when the network made correct and incorrect decisions. You see that lateral is relatively easy to predict.

Now change the 'Number of training cycles' up to 50... for example and you'll see performance go up. You've just trained your first neural network!

:::success
100% accuracy

You might end up with a 100% accuracy after training for 50 training cycles. This is not necessarily a good thing, as it might be a sign that the neural network is too tuned for the specific test set and might perform poorly on new data (overfitting). The best way to reduce this is by adding more data or reducing the learning rate.
:::


## Classifying new data

From the statistics in the previous step we know that the model works against our training data, but how well would the network perform on new data?

Click on **Live classification** in the menu to find out. Your device should (just like in step 2) show as online under 'Classify new data'. Set the 'Sample length' to 5000 (5 seconds), click Start sampling and start doing movements. Afterwards you'll get a full report on what the network thought that you did.


If the network performed great, fantastic! But what if it performed poorly? There could be a variety of reasons, but the most common ones are:

* There is not enough data. Neural networks need to learn patterns in data sets, and the more data the better.
* The data does not look like other data the network has seen before. This is common when someone uses the device in a way that you didn't add to the test set. You can add the current file to the test set by clicking ⋮, then selecting Move to training set. Make sure to update the label under 'Data acquisition' before training.
* The model has not been trained enough. Up the number of epochs to 200 and see if performance increases (the classified file is stored, and you can load it through 'Classify existing validation sample').
* The model is overfitting and thus performs poorly on new data. Try reducing the learning rate or add more data.
* The neural network architecture is not a great fit for your data. Play with the number of layers and neurons and see if performance improves.

**As you see there is still a lot of trial and error when building neural networks**, but we hope the visualizations help a lot. You can also run the network against the complete validation set through 'Model validation'. Think of the model validation page as a set of unit tests for your model!

---

## Deploying back to device

With the impulse designed, trained and verified you can deploy this model back to your device. This makes the model run without an Internet connection, minimizes latency, and runs with minimum power consumption. Edge Impulse can package up the complete impulse - including the signal processing code, neural network weights, and classification code - up in a single C++ library that you can include in your embedded software.

To export your model, click on "Deployment" in the left menu, select the proper "library", under 'Build firmware' select your development board, and click at the bottom of the page.

In our case, we just choose Arduino library.



Clicking the **Build** button will export the impulse and build a binary that will run on your development board in a single step. After building is completed you'll get prompted to download a binary:


Save this on your computer. In our case the generated file is:
> `ei-mlwithwio-arduino-1.0.1.zip`.

To deploy it to the Wio Terminal, you have to first:
1. add in the Arduino IDE the Wio Terminal board, see: https://wiki.seeedstudio.com/Wio-Terminal-Getting-Started/.
2. extract the `ei-mlwithwio-arduino-1.0.1.zip` archive and place it in the Arduino libraries folder.

Finally, open Arduino IDE and open the shared project `movements_sense`:


and:




### How about using the smartphone

If we chose the "Smartphone" in the Run your impulse directly:


we get this QR code that will replicate the application in the smartphone... give it a try!!

You should get something like this:

{%youtube Pu2vzFJihSk%}

---
---