# Emotion Recognition

## Preparation

### Requirements

We build the Docker image based on `pytorch/pytorch:1.7.0-cuda11.0-cudnn8-runtime` docker image.

The additional dependencies are contained in `requirements.txt`.

### File Structure

Extract the dataset and checkpoint files to `./data/`, so the file structure becomes:

.
+-- data
|   +-- affectnet
|   |   +-- all_data
|   |   |   +-- 1
|   |   |       ...
|   |   |   +-- 1349
|   |   +-- data_training_all.npy
|   |   +-- data_validation_final.npy
|   +-- checkpoints
|   +-- ctbc
|   |   +-- all_data
|   |   |   +-- Anger
|   |   |       ...
|   |   |   +-- Surprise
|   |   +-- fold1_training.npy
|   |       ...
|   |   +-- fold5_validation.npy
|   +-- logs
+-- datasets
+-- models
+-- scripts
+-- utils
+-- CTBC_baseline_train.py
+-- CTBC_hierarchical_train.py
+-- README.md

All files except `./data/` will be copied when building the Docker images, so the `./data/` folder should be mounted manually when running the Docker container.

The trainer will save the latest checkpoint and the checkpoint with the best accuracy to the `./checkpoints/`. The trainer also logs loss and accuracy values to a logfile and TensorBoard files to the `./logs`.

## Command Line Arguments
Several important hyperparameter setting that can inputed from command line:

--arch = model architecture (default: mobilenet)
--dropout = enables dropout before classifier layer
--mask_version = mask type for hierachical model
--dataset = dataset selection (default: affectnet)
--fold = which cross-validation fold, unused for affectnet (default: None)
--batch_size = batch size (default: 32)
--n_class = number of class (default: 7)
--n_epochs = number of epochs
--pretrain = model pretraining (default: imagenet), if custom, specify path to the checkpoint using --pretrain_path
--pretrain_path = path to checkpoint for pretraining
--class_balanced = use Class Balanced Loss
--beta = hyperparameters for Class Balanced Loss (ignored if no --class_balanced flag)
--data_root = dataset directory
--checkpoint_root = path to checkpoint
--log_root = path to logs

## Training

For example, to train MobileNet model on AffectNet for both 7 and 8 classes, run:

./scripts/train_affectnet_baseline_mobilenet.sh

## Evaluation

For example, to evaluate trained MobileNet model on AffectNet for both 7 and 8 classes, run:

./scripts/eval_affectnet_baseline_mobilenet.sh

# IOS Application

## Preparation

### Requirements
We build the application on macOS Catalina

The additional dependencies are contained in `requirements.txt`.
### File Structure

+-- Swift_App
|   +-- convert_model
|   |   +-- image_test
|   |   |   +-- test_image.jpg
|   |   +-- models
|   |   +-- output
|   |   +-- pytorch_model
|   |   |   +-- checkpoint_best.pth
|   |   +-- scripts
|   |   +-- coreml.py
|   +-- App
|   |   +-- emotion
|   |   |   +-- emotion
|   |   |   +-- emotion.xcodeproj

pytorch model should be mounted to pytorch_model folder manually.

The coreML model will save in `./output/`.

## Command Line Arguments
Several important hyperparameter setting that can inputed from command line:

--arch = model architecture (default: mobilenet)
--type = baseline or hierachical model type (default: baseline)
--dropout = enables dropout before classifier layer
--n_class = number of class (default: 7)
--mask_version = mask type for hierachical model 
--testing_model = enabled for testing
--img_path = path to image
--checkpoint_root = path to checkpoint

## Convert Pytorch Model to CoreML model
To convert the model(should be run on macOS), run:

./scripts/pth2coreml.sh

## Run App
* Open the emotion.xcodeproj using Xcode on macOS.
* Run and build the code to the device

# Update for subset evaluation

Move the `subset.npy` to ctbc folder as shown in below

.
+-- data
|   +-- affectnet
|   |   +-- all_data
|   |   |   +-- 1
|   |   |       ...
|   |   |   +-- 1349
|   |   +-- data_training_all.npy
|   |   +-- data_validation_final.npy
|   +-- checkpoints
|   +-- ctbc
|   |   +-- all_data
|   |   |   +-- Anger
|   |   |       ...
|   |   |   +-- Surprise
|   |   +-- fold1_training.npy
|   |       ...
|   |   +-- fold5_validation.npy
|   |   +-- subset.npy <---------|
|   +-- logs                     |
+-- datasets                     |
+-- models                       |
+-- scripts                      |
+-- utils                        |
+-- CTBC_baseline_train.py       | 
+-- CTBC_hierarchical_train.py   |
+-- README.md                    |
+-- subset.npy-------------------|

## Evaluation

To evaluate the subset, run:

./scripts/eval_ctbc_hierarchical_subset.sh