# ML report

簡報講稿 [link](https://hackmd.io/sqfElB69ToGbbq_tHCLXLA?edit)
[簡報](https://www.canva.com/design/DAFVjnB-BBI/PWRX2_IrnmRzEo2RIKzjkQ/edit?utm_content=DAFVjnB-BBI&utm_campaign=designshare&utm_medium=link2&utm_source=sharebutton)

## abstract(簡介研究內容，不能提到後面正文沒提到的)
This research subjects mainly focus on biologically Asian males in the age between 18 to 40 years old. We collect some clear front face photos of our subjects(inculding homosexual and heterosexual) as input data, and these data sources are coming from public figures and dating apps. Then in our model, online pre-trained facial recognition model used, and with our own design added, it could be able to classify into two groups by facial features(e.g. noses and eyes).

## Introduction(人臉辨識的歷史脈絡)
There is a study from Stanford University which can correctly distinguish between straight and gay, it's accuracy can be as high as 81% for men and 74% for girl. They used deep neural networks based on a sample that have more than 35,000 facial images to complete this model.

The purpose of this thesis is to apply VGG Face and SVM to detect sexual orientation.
The reason why we want to study on this topic is that we hope our research can help people to reduce the probability of falling in love with men with different sexual orientation.

VGGFace is a face recognition model. It is trained on VGGNet’s own dataset, which contain 2.6 million face images of 2,622 people that is used development face recognition technology, recognizing who is the person in a photo among this dataset. Most of the recent research use it to recognize whether two photos are the same person or their similarity. And SVM is a relatively simple Supervised Machine Learning Algorithm, used for classification and/or regression. It will finds a hyper-plane that creates a boundary between the types of data.

Therefore, we decide to combine the distinguishing ability of VGGFace and the classification ability of SVM as our structure of model.

## Method
### data collection
- Training / validation images
We select photos among biologically Asian males in the age between 18 to 40 years old. Photos we collect mainly from public figures and dating apps, and some from our friends with their consent.
How we label the images is checking if the person got married, if he have a boy/girlfriend, or the label he added in the dating app. Moreover, we labeled people who are bisexual as homosexual, too.
Besides, in order to ensure impartiality, for each person who is in our training data, other photos of him will not be picked as validation data.

We originally collect 390 training images in total, and 181 validation images in total.
- Testing images
Four pictures for men who volunteered to be our testing data.

### data preprocessing
- Detecting Faces and Cropping and Images
Since our input data are general images, which may contain not only male faces but also other unrelated objects, in order to focus on the data of our interest, we must first locate the faces and crop only those part of images. To this end, we use face detector of Dlib for face detecting. This face detector will based on the number of detected faces and each face's scale to determine the sizes to plot the images. Then according to the acquired image size to construct a blank image, which is an array, and then start to fill the array with the human face data. Since the cropped photos are in different sizes, we resized these photos to 224x224, which is the format VGGFace required.

- Data Augmentation
In order to increase the amount of training data and avoid the problem of over fitting, we augment the data before putting it into model. The methods we use are shown bellow:
1. Gamma: tf.image.adjust_gamma()
Perform gamma correction on the input images to adjust the brightness; we set the parameter = 0.2.
3. Contrast: tf.image.adjust_contrast()
Adjust contrast of the input images; we set the parameter = 8.
5. Horizontal Flip: tf.image.flip_left_right()
Flip the images horizontally(left and right).
7. Saturation: tf.image.adjust_saturation()
Adjust saturation of the input images; we set the parameter = 0 and 1.5, which thus produces 2 batches of data.
8. Hue: tf.image.adjust_hue()
Adjust hue of the input images; we set the parameter = 0.05.

Last, we will change the dataset from 390 images to 2730 images.

gay:868
straight:1862
val gay:77
val straight:104

### Model 1 VGGFace+SVM
- VGG Face
VGGFace is a face recognition model, which is pre-trained on VGGNet by their own dataset. We use VGGFace to extract the features of the human facial features from the images.
We download the VGGFace package from [這篇文章的code].
The model will first, transform the training data into the readable format for VGGFace, and input the data into the model. Then, we will acquire the features, which can represent the important infomation of the faces.

- SVM
SVC, which is an application of SVM for classification. Since we need to classify people into two groups by their sexual orientation, we use sklearn.svm.SVC() to train the model.

- ROC curve & best threshold
By plotting and analyzing the ROC curve, we found out the best threshold value for our model is 0.648.

- Model Attributes
We use VGGFace as our base model to extract facial features in the images, and train them with SVM in turn.
Below are some attributes of our VGGFace model.
* weights = vggface
We set the weights as 'vggface', which is a set of pretrained weight by vggface.
* include_top = True
We set include_top to True to include the 3 fully-connected layers at the top of the network.
* model = vgg16
There are three base model choices for VGGFace, which are vgg16, resnet50 and senet50. Eventually, we chose vgg16 for optimization.

- Layers Structure
We extracted the facial features from layer 'fc7' in VGGFace model, and trained them with SVM to classify and generate the result.

### Model 2 VGGFace+NN
- VGG Face
We use the same VGGFace model as VGGFace+SVM one, and acquire the features from layer 'fc7', which represent the important information of the faces.

- NN
Since we need a model which can find the relationship between data automatically, we use NN to train the model. The reason why we didn't choose CNN is that the input data of our model are features, not pictures.

- ROC curve & best threshold
By plotting and analyzing the ROC curve, we found out the best threshold value for our model is 0.845.

- Model Attributes
We use VGGFace as our base model to extract facial features in the images, and train them with NN in turn.
The attributes of our VGGFace model are same as the previous one.

- Layers Structure
Same as the previous model, we extracted the facial features from layer 'fc7' in VGGFace model, and trained them with NN to classify and generate the result.
* optimizer = 'adam'
We have chosen 'adam' as our optimizer in our VGGFace+NN model and set the learning rate to 0.001.
* Loss function = Binary Cross Entropy
Because our classification is bipartite, the binary cross entropy is more suitable to evaluate our model.
* Activation = Relu
* Drop out rate = 0.25
In order to prevent overfitting, we added a drop out layer after the first activation function.
* Activation = Relu
* Activation = Relu
* Drop out rate = 0.5
In order to prevent overfitting, we added a drop out layer.
* Activation = Relu
* Activation = Sigmoid
* Epoches = 50

## Result

Following are our prediction :

- Original Label :
......貼圖片
Fig.??

* VGG+SVM Model Predict Result :
......貼圖片
Fig.??
We got one wrong prediction on heterosexual, and one wrong prediction on homosexual respectively.

* VGG+NN Model Predict Result :
......貼圖片
Fig.??
We got one wrong prediction on heterosexual, and all correct prediction on homosexual respectively.

The accuracy is roughly 75%.

Fig.??

Fig.??

## Discussion and Conclusion
- Result Analysis
1. We have trained three models, including VGGFace+SVM, VGG+NN, pure CNN models.
2. We compare the performance of the VGG-Face+SVM model, VGG-Face+NN model and the CNN model from homework 4, and find that the models using VGG-Face perform much better than the CNN model. In addition, the performance of model using SVM is equal to the performance of model with NN.
3. In the confusion matrix of our VGGFace+SVM model, we observed that the percentage of gay misclassified as straight is a bit higher than the percentage of straight misclassified as gay. The main reason for this may be attributed to the little imbalance in our training data, since the numberof our straight images is slightly more than gay ones in training data.

- Difficulty and Solution
1. data not enough
Since we collect training data, validation data, and testing data all on our own instead of using readily available data, it is hard for us to collect enormous amount of input data. Furthermore, many people don’t easily express their sexual orientation to the others, not to mention to expose this private information clearly on the internet, which makes our data-collecting job even harder and hence, of course, the less amount of our input data. To this end, to solve this problem, we perform augmentation on our original images and increase the amount from 390 to 2730 in total consequently, since data augmentation can not only be the remedy of insufficient data but also a method which can regularize the learning process to prevent the overfit.

2. VGG model application adjustment
During the training process, we observed very strange behavior of our loss function, and acquired a not very good accuracy of the data at the end. Digging into the problem, we surprisingly found that our model predicted all men as straight, which is obviously wrong result. We then went back to restudy VGG Face model once again, finding out that the output data of VGG Face model is actually based on how similar the input image is to the people from its original training dataset (i.e., VGG-Face Dataset, which contains 2622 different public figures’ face images) rather than our own input data set. To solve this problem, we modified our model: instead of running all layers of VGG Face model, we only extract human facial features generated from one of its layers, "fc7". Then, we input the extracted facial features into the newly added SVM, calculating the ROC curves to estimate the performance of the classification and improve our model with the best threshold.

- Future Work
To improve the accuracy and practicality of our model.We will collect more input data to generalize the model and do further study on adding bisexual tag, classify female edition, or even expand our the classifying range of our model to all kinds of races.
## Author Contribution Statements
109060013 Hsin-Yu Chang 章魚(16.6%)
Data collection, Model designing, report writing

109060016 Ting-Ju Lin 林亭汝(16.6%)
Data collection, Image preprocessing, report writing

109060024 Yu-Yan Huang 黃宇妍 (16.6%)
Data collection, Model designing, presentation, report writing

109060029 Chia-Chen Chiang 江(16.6%)
Data collection, Image preprocessing, Model designing, report writing

109060036 Yi-Hua Lin 林宜樺(16.6%)
Data collection, Model designing, report writing

109060037 Man-Dy Yen 顏曼娣(16.6%)
Data collection, Image preprocessing, report writing

## References
https://www.cnblogs.com/adaminxie/p/8339863.html
https://paperswithcode.com/dataset/vgg-face-1
https://github.com/rcmalli/keras-vggface
https://www.cc.ntu.edu.tw/chinese/epaper/0043/20171220_4306.html
https://www.theguardian.com/technology/2017/sep/07/new-artificial-intelligence-can-tell-whether-youre-gay-or-straight-from-a-photograph