# Tutorial: Fire monitoring camera
#### Project in the course 1DT305 Applied IoT by Eliz Fikret - ef223hw

This project initially started as a simple face detection system, but was adjusted to match the capabilities of the chosen microcontroller and the limitations of Mycropython as well as the time limations of the course. In it's current state it could be used as a base for an interesting fire and thermal accident monitoring IOT System, however some adjustments have to be made.
This is quite a simple project, which should take about 5-10 hours, depending on prior knowledge in the area.

## Objective

The reason I chose this project was initially the fact that I got interested in the field of computer vision. I really wanted to work with a camera. Moreover, I found out about the ESP32-Cam Module, which was extremely affordable and I also got very curious to test its capabilities.


As I explained earlier, the project was intented to be a face-recognition system, however some problems were faced on the way. There are many libraries for Python, which make it possible to build face detection systems relatively quickly. On the other hand, I found out that Micropython was very limited in this area. At least I could not manage to run OpenCV on MicroPython to be able to make the face detection work.

Because of this, once I found out that the ESP32-CAM module also has a built-in temperature sensor, I thought of a new idea. The camera together with the temperature sensor could be used to monitor for high temperatures and take pictures to report potential fire. I can imagine this could be useful in a manifacturing setting or even at home for monitoring if an electrical applience which can cause fire has been forgotton.

## Material

| Material | Cost | Description |
| ----------------- | ---- | ----------------------------------------- |
| ESP32-CAM | ~€9 | The main device/microcontroller,which comes with a camera and a temperature sensor |
| USB-Cable | ~€4 | Used for powering the device |
| Jumper Wires | <€1 | Used to connect ESP32-CAM to USB-TTL Converter |
| USB-TTL Converter | ~€2 | Used to be able to plug the device to USB |

All bought from local bulgarian shop for electronic components called Elimex.

## Computer setup
I have used Thonny, because of its simplicity. Also I didn't need the pymakr plugin from Atom as I am not using the Pycom devices.

### Some prerequisites
1. I am using a macOS(Montery) Operating system for this tutorial. Some parts of it like checking the usb ports might be different for Windows or Linux.
2. Install Python
You need to have Python installed on your computer for this tutorual to work. You can go to https://www.python.org/downloads/ for this.
3. Install Node.js
Also, install Node.js if you haven't already. You can use this link: https://nodejs.org/en/

### Connecting the ESP32-CAM to USB-TTL adapter

Since the ESP32-Cam module does not come with a USB port, in order to power it I had to first connect the module to a USB to TTL adapter. The schematic can be seen on the image above. The green connection from IO0 to GRN is needed there only when flashing the device and it should be disconnected afterwards.

| Device Pin | USB to TTL Pin |
| ---------- | -------------- |
| GND | GND |
| 5V | 5V |
| UOR | TX |
| U0T | RX |

### Flashing the firmware
With Python 3 installed, using the Terminal install esptoolpy with pip:

pip install esptool

For this tutorial you will need to flash a custom firmware, which includes the camera functionality. It can be found [here](https://github.com/lemariva/micropython-camera-driver) under the firmware folder. I installed the file ending with 'ble-camera.bin'.


Open Thonny. In order to flash the devide go to Run/Select an Interpreter. Select your Port and click 'Install or update firmware'. Then select the correct firmware and click install. Keep the GPIO 0 to GND connected while flashing the firmware and remove it once you are done or when it starts writing the code.
If you don't know, which your port is check the section below to read how you can find it, at least on Mac.

### Finding the Serial Port
To get a list of active serial ports on Mac you can write the following code in your Terminal.

ls /dev/cu.*

### How the code is uploaded

## Putting everything together

I decribed how I have connected the adapter to the board in the previous section. I have not used any additional sensors except the built-in temperature sensor and the camera that come with the ESP32-Cam Module.
Here I could mention that this is obviously a setup in progress and not ready to be used for development as it an also be seen on the image below.

## Platform
Initially I chose to create a local server to which I could send the pictures taken from the camera and visualize them on an html page. This worked great for the purpose.

However, I also wanted to experiment with using a dashboarding cloud platform for visualizing the temperature of the module. For this purpose I chose Adafruit. Adafruit allows for monitoring data with the use of Feeds and visualizing it within the dashboard feature.

I also experimented with sending the images here directly to this dashboard instead of my local server, but I did not manage, because the images had to be in base64 format specifically. I would say this is a limitation I faced with Adafruits and if I had more time I would try a different platform for this.

## The code

The full code can be found in [this GitHub repository](https://github.com/elizfikret/IOT).

It consists of 4 python files, an image file and an html file where the server is accessed.

Within the main.py file we are connecting to wifi. Then the camera is initialized and some extra configurations for the camera are defined. Afterwards, the local server is created. When the index.html page is refreshed the device takes a photo and saves it on the image.jpg file, which is linked to the intex.html file. The html file needs to include an image tag there the image.jpg file is attached. Don't forget this!

#### The connection to the internet happens in wifiConnect, which is linked to the main.py file

def connect():
  ssid = "your wifi name"
  password =  "your wifi password"
 
  station = network.WLAN(network.STA_IF)
 
  if station.isconnected() == True:
      print("Already connected")
      print(station.ifconfig())
      return
 
  station.active(True)
  station.connect(ssid, password)
 
  while station.isconnected() == False:
      pass
 
  print("Connection successful")
  print(station.ifconfig())

#### The camera is initialized within this code and the picture is taken within the take_photo() function

try:
camera.init(0, format=camera.JPEG)

except Exception as e:
camera.deinit()
camera.init(0, format=camera.JPEG)
camera.flip(1)
camera.mirror(1)
camera.brightness(-1)
camera.quality(6)

def take_photo():
#Take a photo
print("taking picture")
camera.framesize(camera.FRAME_240X240)
buf = camera.capture()
f = open('image.jpg','w')
f.write(buf)
f.close()

#### The following snippet creates a web server on the device and port 80 used for HTTP
For the code for handling the request and response check the main.py file within the Github repo.

#Setup the Web Server
#AF_INET - use Internet Protocol v4 addresses
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('',80))

#### This is how the temperature is retreived from the build-in temperature sensor

esp32.hall_sensor() # read the internal hall sensor
temp = esp32.raw_temperature() # read the internal temperature of the MCU, in Fahrenheit
esp32.ULP()
print(temp)

## Transmitting the data/ connectivity

The wireless protocol used in this project is my local wifi network. As shown in the previos section a seperate file was created with the wifi credentials, which was imported to the main file.

The transport protocols used are for loading the images to the html page in my local server is a webhook (HTTP). Below the code for procesing the requesting and building a response is shown. The final response is the content of the index.html file.

while True:

take_photo()

s.listen(80) #listen to incoming requests on port 80
conn, addr = s.accept()
print('got connection from %s' % str(addr))

#process the requested filename
request = conn.recv(1024)
request = str(request)
string_list = request.split(' ')
#used to be sring_list[0] I changed it
method = string_list[0]
requesting_file = string_list[1]

myfile = requesting_file.split('?')[0]
myfile=myfile.lstrip('/')

if(myfile == ''):
myfile = 'index.html'
print("myfile",myfile)
try:
#Serve up the file
file = open(myfile,'rb')
print("file",file)
response = file.read()
file.close()

# build a header response
header = 'HTTP/1.1 200 OK\n'
if(myfile.endswith(".jpg")):
mimetype = 'image/jpg'
elif(myfile.endswith(".css")):
mimetype = 'text/css'
else:
mimetype = 'text/html'
header += 'Content Type: ' + str(mimetype) + '\n\n'
except Exception as e:
header = 'HTTP/1.1 404 Not Found\n\n'
response = '<html><body><center><h3>Error 404: File not found</h3></center></body></html>'

final_response = header.encode('utf-8')
final_response += response
try:
conn.send(final_response)
except:
print("there was an error, resetting")
conn.close()

The transport protocol for sending the temperature data to Adafruit is MQTT. For this the code from the following tutorial was used: :https://hackmd.io/@lnu-iot/ByPRkQTF9

The temperature data is sent every 5 seconds to the Adafruit temp_feed feed.

timer = Timer(0)
timer.init(period=5000, mode=Timer.PERIODIC, callback=send_data)

## Presenting the data

Within Adafruit to present the temperature data different types of blocks were used. I have used the gauge and the line chart visualizations to display the current temperature and the temperature based on time of the day respectively.
To visualize this data within this dashboard I have linked both blocks to the temp_feed, where I send the temperature data.

I also have the image block there, but I didn't manage to present the image there yet and for now I am showing it on my local server.

The dashboard can be seen below:

As seen within the temp_feed the data is sent to the feed and to the dashboard every 5 seconds.


## Finalizing the design

Learning about IOT and playing around with microcontrollers and sensors was a new and exciting experience for me. Even though my project changed a lot from the initial plan, I still enjoyed the journey.


Currently what works is that every time the local server link is refreshed a new photo is loaded on the page. Meanwhile temperature data is sent to Adafruit dashboard every 5 seconds. If I had more time I would show the temperature together with the image in the same place and send a notification to the user when the temperature is dangerously high.

On the gif below the final result can be seen where I make it warmer around the sensor with a lighter and the temperature in the dashboard increases.