---
# System prepended metadata

title: Project Description

---

# Project Description

## Sections
* Introduction/Background
* Objective
* Project
    * Human Pose Estimation
    * Data
        * Collecting data
        * processing
    * Model
    * Application
* Results
* Moving Forward/Conclusion/Future Research
* References

### Introduction/Background
* Introduce the problem statement.
    * Modify from the slides (Presentation to Jonathan)
* Current research - literature review
    * Using CNN (use the whole video frame as input)

### Objective
* To build a system that translates emergency Auslan signs in real-time;
* To explore the robustness of using human pose estimation in complicated tasks. 
* [kiv] bring forth the awareness of the Auslan community towards the public;
* [kiv] integrate our knowledge in digital signal processing and machine learning to build a real-world application

### Human Pose Estimation
* What is human pose esitmation?
    * Human pose estimation is the ability for computers to track human joint keypoints from a video frame.
* How did you do pose estimation?
    * Used OpenPose, open-source software to transform video into a sequence of human keypoints.
    * Well cited and developed by CMU's research team.
    * Contains ability to track:
        * Pose (Skeleton)
        * Hands
        * Foot
        * Face
* How does OpenPose work in relation to related work?
    * Link to Paper: https://arxiv.org/pdf/1812.08008.pdf
    * Multi-Person 2D Pose Estimation using Part Affinity Fields
    * Two approaches in world of human pose estimation:
        * Top Down
            * Find bounding box of human in frame
            * Detect each keypoints directly from the boudning box
            * We see, the number of humans grow linearly with the computational power needed to perform human pose estimation.
        * Bottom Up - OpenPose Way
            * Detect the keypoints of all possible humans first in a frame using Part Affinity Fields.
            * PAFs - a representation consisting of a set of flow fields that encodes unstructured pairwise relationships between body parts of a variable number of people
            * Use a greedy algorithm to then determine which joints are linked (NP-Hard problem)
            * From linked joints, we have inferences of human skeletons in frame.
* How does openpose work?
    *  System takes in color image of size w x h
    *  Present an explicit nonparametric representation of the keypoint association that encodes both position and orientation of human limbs.
    *  Design an architecture that jointly learns part detection and association.
    *  Demonstrated that a greedy parsing algorithm is sufficient to produce high quality parses of body poses, & preserves efficiency regardless of number of people.
    *  Produces 2D locations of anatomical keypoints for each person in image
    
### Data

#### Collecting Data
* Recorded ourselves doing signs, as we lacked proper videos online as our dataset to train our model.
* Synthetic Data Generation:
    * Created scripts of video augmentation to synthetically generate "more" data to train our model.
    * Eg: Frame Flipping, Changing Video Speeds, Frame Translation/Tilting.
* Passed these recorded videos into OpenPose. 
* Then, we labelled data respectively using scripts.
#### Processing Data
* Normalization - having all keypoint x,y within same range (-1,1) 
    * Increase training rate
    * Prevent exploding gradient/vanishing gradient problem? (unsure)
    * Normalized using frame width, height

* K-Fold Cross-Validataion
    * Splitting dataset into train & validation in K different classes.

### Model
* We have data now, so what's next?
    * We need to perform "feature extraction",
    * Determining what/how to format the "data" going into our model.
    * We did this using human pose estimation. 
* What model did you use to recognise signs?
    * Once we have keypoints from Openpose, we can formalise the problem as time series classification.
    * Given a list of data in time, what category is it?
    * We realised from research that LSTM suited the best for this.
    * Our model is based on LSTM, which is a improved RNN specialised in sequential data processing
* What did you do after training the model?
    * We stored checkpoints of the trained model for deployment and re-training?
    * Hyperparameter tuning, we had to re-train and retuned our model for different hyperparameters to get the best performance.

### Application
* What is your application?
    * We wanted to create a web application that recognises signs in real time hosted on MSE-IT's Computers.
    * It involves a web server hosting our model and when people connect to the application, they will interact with the model to have their signs recognised.
* What tools/software systems did you use for the system?
    * Video streaming is based off on WebRTC, the same software framework used for Zoom, Whatsapp calls and more.
    * Model deployment is done using Tensorflow.js
    * OpenPose runs in python, to translate WebRTC video stream to keypoints in real time.
    * Web Framework - Aiortc, a python based web server framework for WebRTC.

### Results
* Model Accuracy
    * Based on our own test set, we achieved x % accuracy of recognising 6 unique Auslan signs.
* Model Recognition Time:
    * [Offline Prediction] Takes up to x seconds to infer.
    * [Online Web App] Takes up to x seconds to infer.
    * [Online Desktop] Takes up to x seconds to infer.
* Able to sustain x amount of real-time connections on server.
* Proved that it is possible to use Pose Estimation to do Sign Language Recognition.


### Moving Forward/Conclusion/Future Research
* Sequence2Sequence detection
    * Transforming a sequence of translated signs into a sequence of words that form a sentence. (Encoder/Decoder Model)
* Port applicaton to systems that stream from CCTVs/IP Cameras to recognise emergencies.
* Increase number of signs detected using an opensource way of collecting data? (Setting up a video stream online where users can record themselves to add signs to the model themselves).
