## ITSC Office Entry Application

### The Camera/Monitor Panel Box

#### 1. Components:
- Monitor: 5 inch with HDMI interface, mini-USB power, mini-USB touch control
- Camera: Logitech 610 USB Webcam
- Thermal Array: MLX90640 32x24 IR array, 55^o x 35^o FOV, I₂C, 3.3V, -40^oC to 300^oC
- Controller: Teensy 4.0 with header, 600MHz ARM Cortex-M7, 1MB RAM, GPIO w/ USB & I₂C

#### 2. Internal Connection:
    IR Array to Teensy: I₂C (4 wires lead cable)

#### 3. External Connection:
    Monitor:    (1) Mini-USB Power,
             (2) Mini-USB Touch Control to External Odroid XU4 SBC
    Camera:     (3) USB to External Odroid XU4 SBC
    Teensy:      (4) USB connecting to External Odroid XU4 SBC for serial transfer and              power

## Internal Control Box
- Internal Control box inside the office contains:
- Odroid XU4 SBC
- 5V 4A Power Supply
- USB Relay
- 12 VDC Power Supply
- SBC Software (based on 4.2)
- read_thread pass the detected face location to heat_thread protected by a mutex lock. Liveness detection is obtained from heath_thread.
- heat_thread
- Display software

### 1. Kiosk Set Up for Odroid XU4 Running Ubuntu
1. Perform uodate:
`sudo apt-get update && sudo apt-get upgrade`

2. Create account named kiosk:
`sudo adduser kiosk`

3. Add the following line to /etc/sudoers:
`kiosk ALL=(ALL) NOPASSWD: ALL`

4. Install required packages:
`sudo apt-get install -y chromium-browser unclutter xdotool`

5. Set up auto login:
`sudo vi /etc/lightdm/lightdm.conf`
#and add:
[SeatDefaults]
autologin-user=kiosk
autologin-user-timeout=0
user-session=ubuntu
greeter-session=unity-greeter
`sudo mkdir /etc/lightdm/lightdm.conf.d`
`sudo vi /etc/lightdm/lightdm.conf.d/50-myconfig.conf`
#and add:
[SeatDefaults]
autologin-user=kiosk

6. Set up kiosk.sh.script
`sudo mkdir /home/kiosk/.config/autostart`
`sudo nano /home/kiosk/.config/autostart/kiosk.desktop`
#and add:
[Desktop Entry]
Type=Application
Name=Kiosk
Exec=/home/kiosk/kiosk.sh
X-GNOME-Autostart-enabled=true<br>
Vi /home/kiosk/kiosk.sh

#!/bin/bash
 
# Run this script in display 0 - the monitor
export DISPLAY=:0
 
# Hide the mouse from the display
unclutter &
 
# If Chromium crashes (usually due to rebooting), clear the crash flag # so we don't have the annoying warning bar
sed -i 's/"exited_cleanly":false/"exited_cleanly":true/' /home/kiosk/.config/chromium/Default/Preferences
sed -i 's/"exit_type":"Crashed"/"exit_type":"Normal"/' /home/kiosk/.config/chromium/Default/Preferences
 
# Run Chromium and open tabs
/usr/bin/chromium-browser --window-size=1920,1080 --kiosk --window-position=0,0 http://google.com http://bing.com &
 
# Start the kiosk loop. This keystroke changes the Chromium tab
# To have just anti-idle, use this line instead:
# xdotool keydown ctrl; xdotool keyup ctrl;
# Otherwise, the ctrl+Tab is designed to switch tabs in Chrome
# #
while (true)
do
    		xdotool keydown ctrl+Tab; xdotool keyup ctrl+Tab;
    		sleep 15
done

7. Set kiosk.sh executable:
`chmod +x kiosk.sh`

8. To logout kiosk without reboot:
`sudo killall kiosk.sh && sudo service lightdm restart`

9. To exit kiosk mode (close chromium):
Press the Window Key to start the launcher.
Type in 'terminal' to start the terminal window.
At terminal: `sudo killall kiosk.sh && sudo service lightdm restart` Or press Alt-F4, then `sudo killall kiosk.sh && sudo service lightdm restart`

## Library Gate Entry Application
### 1. Gate Entrance Camera/Monitor Setup
- A camera connecting to an Odroid XU4 SBC is placed at the left-most gate for entry access control.
- The SBC is connected to the Library subnet.
- A touch screen monitor connected to a Raspberry Pi 3B+ running at Kiosk mode can be placed at the gate to show the video stream captured by the camera.
- The browser web page references the link of the stream served by an http service thread at the Camera Handling Program in MJPG format.
- Another http service thread at the Camera Handling Program serves qualified images to the FR Server for face recognition. <br>

### 2. Ciculation Counter Monitor Station
- Live mode: works same as the Gate Monitor
- Review mode: the user may choose a period for reviewing. The staff may browse and compare.

### 3. FR Service Interface
- Face recognized by the FR server
- Message sent from FR server to the browser running at Kiosk Mode at the Monitor SBC via the web socket server.
- The still picture and corresponding EPPN displayed on the monitor
- Message sent to Library IMS Server to open the gate
- Tap on the screen to log the fault in case of false positive.

### 4. Kiosk Mode Set Up at Pi running Raspbian
1. Wait for network at boot (optional):
• Run sudo raspi-config.
• Select Option 3 (Boot Options).
• Select B2 (Wait for Network at Boot).
• Select Yes
• Exit

2. Perform clean up (optional):
`sudo apt-get remove --purge wolfram-engine scratch nuscratch sonic-pi idle3 smartsim java-common minecraft-pi python-minecraftpi python3-minecraftpi libreoffice python3-thonny geany claws-mail bluej greenfoot`<br>
`sudo apt-get autoremove`

3. Ensure up-to-date:
`sudo apt-get update`
`sudo apt-get upgrade`
`sudo apt-get dist-upgrade`

<b>WHY IS 4. MISSING?</b>

5. Install Chrome etc if not there:
`sudo apt-get install unclutter chromium-browser`

6. Auto Chrome startup:
`vi /etc/xdg/lxsession/LXDE-pi/autostart`

#@lxpanel --profile LXDE-pi
#@pcmanfm --desktop --profile LXDE-pi
@xset s off
@xset -dpms
@xset s noblank
@sed -i 's/"exited_cleanly": false/"exited_cleanly": true/' ~/.config/chromium-browser Default/Preferences
@chromium-browser --noerrdialogs --kiosk http://facedev.ust.hk/ws/ws_lib.html --incognito --disable-translate

7. Reboot:
`reboot`

## Operation
### 1. Face Detection Service at the Registration Server
A Detection Daemon Program written in Python must be running in background at the GPU Server to detect face from the streams captured from the 9 Registration Camera and generate embeddings.<br>
Below is a table of files of the Face detection Service for Registration:
|Description|File Name|Remark|
| :--- | :--- | :--- |
|Server:|frtdev or frtpro||
|Home Dir:|/home/py/reg/||
|Insightface lib:|/home/py/insightface/|Common to all services|
|Server Code:|reg<ver>xI.py|where <ver> is the version number|
|Database Code:|DB_Handler_reg.py||
|Monitoring Parent:|pystart|To restart the server upon crash|
|Classifier:|clf.py||
|Config File:|config.txt||
|Initializing Photo:|init.jpg|To warm up the detection|
|Temp Photo Dir:|snapshot/||
|Startup script:|upreg||
|Shutdown script:|downreg or killreg||
|CPU Utilization Script:|cpu||
|View Process Script:|ps||
|A copy of Pythpn3:|p3|For convenient process monitoring|
|Server Monitoring:|chkreg|Monitor Service Crash|
|Photo Dir:|/nfs/photo/|Registered photo for study|
|Usage Description:|usage.txt||
|Crontab:|crontab –l|Setup for regular restart the service|

### 2. Web-based Registration Interface
The Web-based Registration Interface is installed in the web server VM supported with MySQL database. CodeIgniter PHP Framework is used.
|||
| :--- | :--- |
|Server:| facedev.ust.hk or face.ust.hk|
|CodeIgniter dir:| /var/www/frt/|
|JavaScript:| /var/www/frt/js/en.js|
|CSS file:| /var/www/frt/css/enroll.css|
|Controller:|/var/www/frt/application/controllers/en.php|
|View:|/var/www/frt/application/views/en_view.php|
|Model:|/var/www/frt/application/models/enroll_model.php|
|Config:|/var/www/frt/application/config/config.php|
|Database Config:|/var/www/frt/application/config/database.php|
|Email Config:|/var/www/frt/application/config/email.php|
|Database Maint:|http://facedev.ust.hk/phpmyadmin |

<b>ACTUALLY IS enrol.css INSTEAD OF enroll.css<br>AND enrol_model.php INSTEAD OF enroll_model.php</b>

#### 13.3 Face Recognition Service
A Face Recognition Daemon Program must be installed at the GPU Server to serve FRT applications such as Lib Entrance.
||||
| :--- | :--- | :--- |
|Server:|frtdev.ust.hk or frtpro.ust.hk||
|Home Dir:|/home/py/det/||
|Insightface lib:|/home/py/insightface/|Common to all services|
|Server Code:|det<ver>.py|where <ver> is the version number|
|Database Code:|DB_Handler_new.py||
|Monitoring Parent:|pystart|To restart the server upon crash|
|Classifier:|clf_new.py||
|Email Module:|mail3.py||
|Initializing Photo:|init.jpg|To warm up the detection|
|Temp Photo Dir:|snapshot/||
|Startup script:|updet||
|Shutdown script:|downdet or killdet||
|CPU Utilization Script:|cpu|View CPU/GPU utilization|
|View Process Script:|ps|View Service related processes|
|View GPU Checker:|psg|View GPU Checker processes status|
|View Checker:|psc|View Checker processes status|
|A copy of Pythpn3:|p3|Clone of the python3 program|
|Server Monitoring:|chkdet|Monitor Service Crash|
|Photo Dir:|/home/py/Entry/|Entry photo for monitor and study|
|Crontab:|crontab –l|Setup for regular restart the service|

### 4. Cam Handling for Registration
Programs and files are put in the home directory of each pi handling the registration camera. The service is started automatically during boot.
||||
| :--- | :--- | :--- |
|Home Dir: | /home/pi||
|Cam Handling Prog:|m6b.py|version number might be changed|
|Startup Parent:|pistart||
|Frontal Face Model:|haarcascade_frontalface_default.xml||
|Authorization File:|auth||
|Startup:|upcam||
|Shutdown: |downcam||
|Process Status: |ps||
|Auto Statrtup: |/etc/rc.local||

<b>WHERE ARE /home/pi AND /home/odroid/pi?</b>

### 5. Cam Handling for Face Recognition
Programs and files are put in the home directory of each pi handling the registration camera. The service is started automatically during boot.
||||
| :--- | :--- | :--- |
|Home Dir: | /home/odroid/pi||
|Cam Handling Prog:|m7b.py|version number might be changed|
|Startup Parent:|pistart||
|Frontal Face Model:|haarcascade_frontalface_default.xml||
|Authorization File:|auth||
|Startup:|upcam||
|Shutdown: |downcam||
|Process Status: |ps||
|Auto Statrtup: |/etc/rc.local||

### 6. Web Socket Service
The Web Socket Server is for handling instant communication between web browser and the FRT Service Process.
|||
| :--- | :--- |
|Server: |facedev.ust.hk or face.ust.hk|
|Directory:|/var/www/face/py/ws/|
|Program:|ws2.py|
|Startup Parent:|pystart|
|Startup:|upws2|
|Shutdown: |killws2|
|Process Status:|ps|

### 7. Gate Monitor Host
A Web Page installed at the Gate Monitor Web Server is started by the monitor hosting browser to be displayed at the Gate Monitor.
||||
| :--- | :--- | :--- |
|Web Server: |facedev.ust.hk||
|Server Dir: |/var/www/ws/||
|Monitor Host Config:|/etc/xdg/lxsession/LXDE-pi/autostart |at the monitor SBC|
|Web page: |ws4_lib.html or ws4_itsc.html||
|Logo file: | ust.png||
|No-Entry file:| noentry.jpg||

### 8. Counter Monitor Web App
A PC placed at the Lib Counter to review the accuracy of the entry by a web app. The web based monitoring app is installed in the web server.
|||
| :--- | :--- |
|Server:| facedev.ust.hk or face.ust.hk|
|CodeIgniter dir:| /var/www/mon/|
|JavaScript: |/var/www/mon/js/mon.js|
|CSS file: | /var/www/mon/css/mon_new.css|
|Controller:|/var/www/mon/application/controllers/mon.php|
|View:|/var/www/mon/application/views/mon_view.php|
|Model:|/var/www/mon/application/models/enroll_model.php|
|Config:|/var/www/mon/application/config/config.php|
|Database Config:|/var/www/mon/application/config/database.php|
|Email Config:|/var/www/mon/application/config/email.php|
|Database Maint:|http://facedev.ust.hk/phpmyadmin |
|Lib Photo Dir: |/var/www/mon/Entry_Lib/|
|ITSC Photo Dir:| /var/www/mon/Entry/|

<b>ACTUALLY IS enrol_model.php INSTEAD OF enroll_model.php</b>

### 9. Counter Monitor Host
A normal PC running Win 10 with browser support to display the web app. The startup URL for the counter monitor web page is:<br>
Startup: http://facedev.ust.hk/mon/mon.html

### 10. ITSC Cam Handling Host for FR
Programs and files are put in the home directory of each pi handling the face recognition camera. The service is started automatically during boot.
||||
| :--- | :--- | :--- |
|Home Dir: |/home/odroid/pi||
|Cam Handling Prog:|m8b.py|version number might be changed|
|Startup Parent:|pistart||
|Frontal Face Model:|haarcascade_frontalface_default.xml||
|Authorization File:|auth||
|Startup:|upcam||
|Shutdown: |downcam||
|Process Status:|ps||
|Auto Statrtup:|/etc/rc.local||

To detect liveness, Thermal IR Array is connected to a Teensy Micro Controller that runs a C++ program to collect thermal data and send to the Odroid for liveness checking.
|||
| :--- | :--- |
|IDE for IR Array:|https://www.arduino.cc/en/main/software|
|Library for IR Array:|https://github.com/adafruit/Adafruit-MLX90614-Library|
|Teensy Board Add-on:|https://www.pjrc.com/teensy/td_download.html|
|Directory: |c:\Document\Arduino\heat\|
|Program:|heat.ino|

Procedure of editing, compiling and uploading the Arduino code:
* Invoke Arduino IDE
* Open the program source
* Edit the Program
* Click the Save Button
* Choose Tools -> Board -> Teensy 4.0
* Connect the Teensy to the USB port
* Choose Tools -> Port -> The desired USB port
* Click the Verify Button to Compile
* Click the upload Button to Upload to the Teensy
* The compiled code will be save in the EPROM
#### 13.11 ITSC Door Monitor Host
A Pi is required to run in Kiosk Mode to support the Monitor. The Odroid can also be used if it still has spare processing capacity. For information of setting up the Odroid in Kiosk Mode, please refer to Section 10.4.
||||
| :--- | :--- | :--- |
|Web Server: |facedev.ust.hk||
|Server Dir: |/var/www/ws/||
|Monitor Host Config:|/etc/xdg/lxsession/LXDE-pi/autostart |at the monitor SBC|
|Web page: |ws4_itsc.html||
|Logo file: |ust.png||
|No-Entry file:|noentry.jpg||

#### 13.12 The Opt-in System
A VM is required to host the opt-in application.
|||
| :--- | :--- |
|Server:| facedev.ust.hk or face.ust.hk|
|CodeIgniter dir:| /var/www/frt/|
|JavaScript:|nil|
|CSS file:|nil|
|Controller:|/var/www/frt/application/controllers/appreg.php|
|View:|/var/www/frt/application/views/appreg_view.php<br>/var/www/frt/application/views/prerequisite_view.php<br>/var/www/frt/application/views/consent_view.php|
|Model:|/var/www/frt/application/models/appreg_model.php<br>/var/www/frt/application/models/reg_model.php|
|Config:|/var/www/frt/application/config/config.php|
|Database Config:|/var/www/frt/application/config/database.php|
|Email Config:|/var/www/frt/application/config/email.php|
|Database Main:|http://facedev.ust.hk/phpmyadmin |

#### 13.13 Linear Actuator Host
The host used is an Adruino Mega 2560 Microcontroller with Ethernet Shield. Development is done using the Adruino IDE:
|||
| :--- | :--- |
|IDE Download:|https://www.arduino.cc/en/main/software |
|Directory:|c:\Document\Arduino\linearEth\|
|Program:|linearEthUDP.ino|
|Connection:|Connect to Development PC and disconnect Ethernet cable at Ethernet Shield|
|IP:|Make up a unique MAC address, record it, arrange DHCP IP and record it|
|Code Change:|Hardcode the MAC Address at program before uploading|

Procedure of editing, compiling and uploading the Arduino code:
* Invoke Arduino IDE
* Open the program source
* Edit the Program
* Click the Save Button
* Choose Tools -> Board -> Mega
* Connect the Arduino Mega to the USB port
* Choose Tools -> Port -> The desired USB port
* Click the Verify Button to Compile
* Click the upload Button to Upload to the Arduino Mega
* The compiled code will be save in the EPROM