# Pitender

:::success
Shout out to George and all my friends who gave me a lot of help to build this project!
:::

## Overview
The “Pitender” project is an emotion recognition soda fountain. Pitender can sense environment data(temperature & humidity), and also user's emotion during a time interval. User can activate Pitender through a website, Pitender will sense all the data, then recommend a drink for your condition; for example, when Pitender realizes you're sad, it probably serves a drink can lighten you up. After the drink is served, the website will show how much drinks left in the inventory.

## Things used in this project
### Hardware Components
* Raspberry Pi 3 Model B *1
* Intel® Neural Compute Stick 2 *1
* Arducam Noir Camera for Raspberry Pi [B0036] *1
* DHT22 Temperature Humidity Sensor Module *1
* 5V Two-way Relay Module *1
* Transparent hose (2m) *1
* Micro air pump (5V) *2
* 4xAA battery holder *2
* AA battery *8
* Jumper wires
* Mini breadboard
[Optional] portable charger + microusb cable (You can power supply in different ways)

### Software
* Python 3.7
* Open-Vino (4.1.2-openvino)

### Hand tools
* Hand drill with 5mm dowel drill (To drill through bottle caps)
* Soldering iron **(Be careful when you're using it)**
* Glue gun **(Be careful too)**
* Wire stripper
* Screwdriver 0*100mm

## Circuit Diagram
:::warning
Didn't put pi-camera and neural compute stick 2 on the circuit diagram!

Just plug them in your raspberry pi. It's simple!
:::

## How does Pitender really look like

## Implementation :video_game:
:::info
Assume that you've already installed python 3.7 and Open-Vino on your raspberry pi.
:::

### 0. Environment Configuration
Friendly reminder, make sure all your hardware devices work fine, including
1. Pi-Camera
2. Neural Compute Stick 2
3. Air pump
4. dht22 Sensor
5. Raspberry pi (Sometimes, pin on the pi can be broken)
6. Make sure all the devices can get a power supply

**In addition, transparent hose's size have to match both air outlet on pump and bottle caps.**

### 1. Get data from DHT22 sensor
1. Install CircuitPython DHT Library

pip3 install adafruit-circuitpython-dht

sudo apt-get install libgpiod2

2. Test the sensor

import time
import board
import adafruit_dht
 
# Initial the dht device, with data pin connected to:
dhtDevice = adafruit_dht.DHT22(board.D4) //D4 = pin4
 
while True:
    try:
        # Print the values to the serial port
        temperature_c = dhtDevice.temperature
        temperature_f = temperature_c * (9 / 5) + 32
        humidity = dhtDevice.humidity
        print("Temp: {:.1f} F / {:.1f} C    Humidity: {}% "
              .format(temperature_f, temperature_c, humidity))
 
    except RuntimeError as error:
        # Errors happen fairly often, DHT's are hard to read, just keep going
        print(error.args[0])
 
    time.sleep(2.0)

Then Run the script

python3 xxx.py  //xxx is your file name

:anger: As the exception said, dht22 is really hard to read sometimes. Keep running it until it success.

Source: https://learn.adafruit.com/dht-humidity-sensing-on-raspberry-pi-with-gdocs-logging/python-setup

### 2. Get Emotion Data using emotion recognition
0. Make sure you've already done installed OpenVino and plugged your ncs2 and Pi-camera in raspberry pi
1. Clone the project
`git clone https://github.com/PINTO0309/OpenVINO-EmotionRecognition.git`

Then run the script
`python3 main.py`

:smile: Now you can detect user's emotion in real time!
Source: https://github.com/PINTO0309/OpenVINO-EmotionRecognition

### 3. Recommend Drink
You can have your own recommend method! Here I'm gonna share how I recommend drinks.

1. Build Recommend Table

| Dimension | Score |
| ------ | ----------- |
| Temperature(20%) | <25˚C (-1) , >25˚C (+1) |
| Humidity(20%)| <70% (-1) , >70% (+1) |
| Emotion(60%) | anger (1), sad (2), neutral (3) , happy (4) , surprise (5) |
Temperature, Humidity, Emotion account for 20%, 20%, 60% respectively in formula. Number in the Parentheses means the score you get from that dimension; for example, you're sad, temperature is 24˚C and humidity is 71%, your total score is -1* 0.2 + 1* 0.2 + 2* 0.6 = 1.2 .

Score interval is from 0.2 to 3.4, so I divide this interval into four parts, you can decide which drink you wanna recommend. In my case, I can recommend four different kinds of drink.

### 4. Run your application on server
1. Install flask if you don't have it on your raspberry pi, you can check it with command.

python3
>> import flask
>> flask.__version__

or

`pip freeze | grep flask`

2. Build the server

from flask import Flask

app = Flask(__name__)

@app.route('/')             // root route, every route have to return something
def index():
    return 'Hello world'

if __name__ == '__main__':     
    app.run(debug=True, host='0.0.0.0')

`__name__ == '__main__`: if you run this py script, this is the main function. But if you import this into another py script, it won't carry out.

3. Customize it

from flask import Flask

app = Flask(__name__)

@app.route("/")
def start():
    return render_template('main.html')

@app.route("/control/")
def main():
    
    mode_of_camera = 1
    number_of_camera = 1
    camera_width  = 640
    camera_height = 480
    number_of_ncs = 1
    vidfps = 30
    fd_model_path = "./FP16/face-detection-retail-0004"
    em_model_path = "./FP16/emotions-recognition-retail-0003"
    

    mp.set_start_method('forkserver')
    frameBuffer = mp.Queue(10)
    resultsFd = mp.Queue() # Face Detection Queue
    resultsEm = mp.Queue() # Emotion Recognition Queue
    
    # Start streaming
    p = mp.Process(target=camThread,
                   args=(LABELS, resultsEm, frameBuffer, camera_width, camera_height, vidfps, number_of_camera, mode_of_camera),
                   daemon=True)
    p.start()
    processes.append(p)

    
    #get emotion data   
    emotion_data = inferencer(resultsFd, resultsEm, frameBuffer, number_of_ncs, fd_model_path, em_model_path)
    
    # Turn ['string'] into real string
    emotion = ' '.join(emotion_data) 
    
    #Recommend drink
    recommend_drink = RecommendDrink(emotion)
    
    #Output the drink
    makeDrink(recommend_drink)
    
    # capacity: 600ml
    coke_capacity = 600
    sprite_capacity = 600
    
    # 1sec will output 25ml liquid
    if recommend_drink == "Sprite_More":
        coke_capacity-= 25*1.5
        sprite_capacity-= 25*3.5
    
    elif recommend_drink == "Sprite":
        sprite_capacity-= 25*5
        
    elif recommend_drink == "Coke":
        coke_capacity-= 25*5
        
    elif recommend_drink == "Coke_More":
        coke_capacity-= 25*3.5
        sprite_capacity-= 25*1.5
    else:
        print("Something went wrong!!!")
        
    return render_template('inventory.html',recommend_drink=recommend_drink, coke_capacity=coke_capacity, sprite_capacity=sprite_capacity)
    
if __name__ == "__main__":
   app.run(host='0.0.0.0', port=5000, debug=True)

:key: Port number should be 5000, 5000 is a privileged port. If you don't use 5000, you have to run script with `sudo`, which is really bad. Because all your libraries are installed under pip3 environment, not sudo pip3 environment. So use port number 5000 can solve all the problems.

Reference: https://stackoverflow.com/questions/38298652/permissionerror-errno-13-permission-denied-flask-run

Source: https://projects.raspberrypi.org/en/projects/python-web-server-with-flask

## More detailed information

### Demo Video
https://youtu.be/Wb9AC1lJPIE

### Code
https://github.com/ballm06m06/Pitender