# Programming Assignment 1: Hand Pose Recognition with CNN
###### tags: `東華四下` `深度學習`
[TOC]

---
## a) Method descriptions and Hyper Parameters

- 使用pytorch 框架完成本次作業
- CNN架構使用3層convolution，2層pooling，flatten
- layer中pooling:一組cnn是用MAX，一組是用AVERAGE比較其結果
- batch:50個一組
- convolution的kernel size(filter size): 5*5
- pooling的kernel size(filter size):2*2
- stride:每次往右一格
- activation:皆選用ReLU函式
- epoch:有做10次跟20次去比較結果
- learning rate:有設0.001跟0.0001去比較結果

## b) Source code explanations
### 一. 資料預處理preprocessing
#### step1
安裝jupyter pytorch及opencv的環境，並確認自己的電腦有無cuda GPU
結果是沒有
#### step2
將所獲的的圖片檔資料夾分成training data跟test data兩部分
training: SET 1-3
testing: SET 4-5
並將分割好的兩部分檔案分別儲存於各自的csv檔，以便之後使用
#### step3
creating a custom dataset
#### step4
讀入剛剛處理好的csv檔案
### 二. 建立CNN模型
#### step5
建立卷積神經網路模型

### 三. 建立訓練模型
#### step6
定義準確值計算
#### step7
建立訓練模型
#### step8
調整參數，比較結果

## c) Experimental results
### 一. 以下是用max_pool得出之結果
#### ADAM v.s. SGD(LR=0.001,EPOCH=10)

#### ADAM v.s. SGD(LR=0.0001,EPOCH=10)

#### ADAM v.s. SGD(LR=0.001,EPOCH=20)

#### ADAM v.s. SGD(LR=0.0001,EPOCH=20)

### 二. 以下是用average_pool得出之結果
#### ADAM v.s. SGD(LR=0.001,EPOCH=10)

#### ADAM v.s. SGD(LR=0.0001,EPOCH=10)

#### ADAM v.s. SGD(LR=0.001,EPOCH=20)

#### ADAM v.s. SGD(LR=0.0001,EPOCH=20)

## d) Discussions on the results
### 一.結果
- 1.不管是哪一種參數調整，可以發現Adam的loss都比SGD下降還要穩定
- 2.training準確值皆高於testing
- 3.Adam整體效果較佳
- 4.learning rate在0.0001時的loss值比0.001時還佳
- 5.epoch越高時，loss值越低
- 6.pooling設成Max時整體loss較低，效果比AVG好很多
### 二.改進
- 1.可以將adam和SGD合成一張圖，去比較LOSS的變化和下降趨勢(斜率)
- 2.可以修改神經層的變化在去比較其結果，想辦法提升準確率，及loss值降低
- 3.overfitting處理
- 4.嘗試更多種優化器ex.Adagrad

## e) Concluding remarks
這學期剛好有課程也使用到OPENCV的部分以及老師有給許多參考範例，所以在一開始預處理的部分不算花太多時間，真正複雜的是要先了解CNN整體架構是怎麼運作，才有辦法按照自己的想法設計出CNN layer，在設計卷積神經網路時，遇到了如何給MATRIX值，只要上一層連接下一層的大小給錯，就會跑不動，要弄很清楚自己當前的CANNEL大小。在給pattern大小時，一直很不確定自己有沒有抓到適合尺寸。花了不少時間完成這份作業，也因為這份作業讓我更了解深度學習在做些甚麼，從複雜的數學理論，轉換成可見的程式，覺得很有趣，也謝謝老師給予我們充裕的時間完成這份作業，讓我們可以多花一些時間自己上網查資料，學習人家怎麼建構這些神經網路。

## f)Reference

[1.深度學習實戰（九）—— loss不下降的解決方法](https://www.twblogs.net/a/5d175588bd9eee1ede055ad6)

[2.Day 08：CNN 模型設計](https://ithelp.ithome.com.tw/articles/10192028)

[3.菜雞 Pytorch MNIST 實戰 part2](https://ithelp.ithome.com.tw/articles/10224627)

## g)code

#gpu,cpu測試
import torch
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(device)

#spliting將照片分成training data 跟 testing data並建立於csv檔中
import cv2
import numpy as np
import os
import pandas as pd

def enumerate_files(dirs, path='All_gray_1_32_32/', n_poses=3, n_samples=20):
    filenames, targets = [], []
    for p in dirs:
        for n in range(n_poses):
            for j in range(3):
                dir_name = path+p+'/000'+str(n*3+j)+'/'
                for s in range(n_samples):
                    d = dir_name + '%04d/' % s
                    for f in os.listdir(d):
                        if f.endswith('jpg'):
                            filename = d + f
                            filename = filename.replace("All_gray_1_32_32/", "")
                            filenames += [filename]
                            targets.append(n)

    return filenames, targets

def read_images(files):
    imgs = []
    for f in files:
        img = cv2.imread(f, cv2.IMREAD_GRAYSCALE) #使用opencv讀入照片，並轉成灰階影像GRAYSCALE
        imgs.append(img)
    return imgs

#自訂dataset:欄位為filename,label
def read_datasets(datasets, csv_name):
    files, labels = enumerate_files(datasets)
    dataframe = {"filename": files, "label": labels}
    dataframe = pd.DataFrame(dataframe)
    dataframe.to_csv(csv_name)
    list_of_arrays = read_images(files)
    return np.array(list_of_arrays), labels

#將兩個set儲存成csv檔
train_sets = ['Set1', 'Set2', 'Set3']
test_sets = ['Set4', 'Set5']
trn_array, trn_labels = read_datasets(train_sets, csv_name = "train_data.csv")
tst_array, tst_labels = read_datasets(test_sets, csv_name = "test_data.csv")

import os 
import pandas as pd
from torchvision.io import read_image
import torchvision.transforms as transforms
from torch.utils.data import DataLoader
import torch.nn as nn
import torch
from torch.utils.data import Dataset
from skimage import io
import matplotlib.pyplot as plt
import numpy as np 

#建立Dataset
class CustomImageDataset(Dataset):
    def __init__(self,  annotations_file, img_dir, transform=None):
        self.img_labels = pd.read_csv(annotations_file)
        self.img_dir = img_dir
        self.transform = transform
        #self.target_transform = target_transform
        
    def __len__(self):
        return len(self.img_labels)

    def __getitem__(self, index):
        img_path = os.path.join(self.img_dir, self.img_labels.iloc[index, 1])
        image = io.imread(img_path)  
        label = torch.tensor(int(self.img_labels.iloc[index, 2])) 
        if self.transform:
            image = self.transform(image)
        #if self.target_transform:
         #   label = self.target_transform(label)
        return image, label

#讀入預先處理好的csv檔們
train_data = CustomImageDataset(annotations_file = "train_data.csv", img_dir = "All_gray_1_32_32", transform = transforms.ToTensor() )
test_data = CustomImageDataset(annotations_file = "test_data.csv", img_dir = "All_gray_1_32_32", transform = transforms.ToTensor() )
train_dataloader = DataLoader(train_data, batch_size=50,shuffle = True)
test_dataloader = DataLoader(test_data, batch_size=50, shuffle = True)

Padding=int(5/2)

#cnn layer
class NeuralNetwork(nn.Module):
    def __init__(self):
        super(NeuralNetwork, self).__init__()
        #第一層
        #因為讀入的是黑白照片，所以input_channel會為1個頻道 
        self.cnn1 = nn.Conv2d(in_channels=1, out_channels=16, kernel_size=5, stride=1, padding=Padding)                               
        self.relu1 = torch.nn.ReLU()  # activation                     
        self.maxpool1 = torch.nn.MaxPool2d(kernel_size=2)    
        
        #第二層
        self.cnn2 = torch.nn.Conv2d(in_channels=16, out_channels=32, kernel_size=5, stride=1, padding=Padding)                                
        self.relu2 = torch.nn.ReLU()  # activation                     
        self.maxpool2 = torch.nn.MaxPool2d(kernel_size=2)  
        
        #第三層
        self.cnn3 = torch.nn.Conv2d(in_channels=32, out_channels=64, kernel_size=5, stride=1, padding=Padding)                                
        self.relu3 = torch.nn.ReLU() # activation                      
        
        #cnn3完的輸出為64，為linear的input
        self.fc1 = torch.nn.Linear(64 *8 *8, 3) 
        
        #一層一層做傳遞
    def forward(self, x):
        x = self.cnn1(x)
        x = self.relu1(x)
        x = self.maxpool1(x)
        x = self.cnn2(x)
        x = self.relu2(x)
        x = self.maxpool2(x)
        x = self.cnn3(x)
        x = self.relu3(x)
        x = x.view(x.size(0), -1)    #-1表示一个不确定的数 
        output = self.fc1(x)
        return output, x 
        
cnn_model = NeuralNetwork()  #將自訂的cnn_layer命名為cnn_model

#計算準確值
#正確值/樣本數=accuracy
def cnn_accuracy(loader, cnn_model):
    correct = 0
    sample = 0
    cnn_model.eval()

    with torch.no_grad():
        for x, y in loader:
            scores = cnn_model(x)[0]
            _, predictions = scores.max(1)
            correct = correct + (predictions == y).sum()
            sample = sample + predictions.size()[0]
        print(f'Accuracy {float(correct)}/{float(sample)*1} = {round((float(correct)/float(sample))*100, 2)}')
    
    cnn_model.train()

#Adam,learning rate= 0.001
optimizer = torch.optim.Adam(cnn_model.parameters(), lr=0.001)
loss_func = nn.CrossEntropyLoss()          #使用cross entropy                 
print('Adam result:')
print('learning rate: 0.001')
print('epoch: 10')
list_loss = []
list_epoch =[]

#epoch設10
for epoch in range(10):
    losses = []
    for step, (data, target) in enumerate(train_dataloader):  
        
        output = cnn_model(data)[0]         #data丟入所設計的cnn內   
        loss = loss_func(output, target)   #算loss值
        losses.append(loss.item())

        optimizer.zero_grad()        #grad設成0
        loss.backward()               #backward Calculate gradients

        optimizer.step()  #Update parameters           
#輸出
    list_loss.append(sum(losses)/len(losses))
    list_epoch.append(epoch)
    print(f'Epoch {epoch} is {sum(losses)/len(losses)}')
    

print("Accuracy_Training: ")
cnn_accuracy(train_dataloader, cnn_model)
print("-------------------------------------------------")
print("Accuracy_Testing: ")
cnn_accuracy(test_dataloader, cnn_model)
print("-------------------------------------------------")

#畫圖
x = list_epoch
y = list_loss
plt.plot(x, y, 'g-')
plt.title("<loss function>")
plt.xlabel("epoch")
plt.ylabel("loss")
plt.grid(True)
plt.show()

#SGD,learning rate= 0.001
optimizer = torch.optim.SGD(cnn_model.parameters(), lr=0.001)
loss_func = nn.CrossEntropyLoss()                           
print('SGD result:')
print('learning rate: 0.001')
print('epoch: 10')
list_loss = []
list_epoch =[]

#epoch設10
for epoch in range(10):
    losses = []
    for step, (data, target) in enumerate(train_dataloader):  

        output = cnn_model(data)[0]              
        loss = loss_func(output, target)  
        losses.append(loss.item())

        optimizer.zero_grad()           
        loss.backward()   
        
        optimizer.step()              

    list_loss.append(sum(losses)/len(losses))
    list_epoch.append(epoch)
    print(f'Epoch {epoch} is {sum(losses)/len(losses)}')
    
print("Accuracy_Training: ")
cnn_accuracy(train_dataloader, cnn_model)
print("-------------------------------------------------")
print("Accuracy_Testing: ")
cnn_accuracy(test_dataloader, cnn_model)
print("-------------------------------------------------")


x = list_epoch
y = list_loss
plt.plot(x, y, 'g-')
plt.title("<Loss Function>")
plt.xlabel("epoch")
plt.ylabel("loss")
plt.grid(True)
plt.show()