Pytorch Tainan 090118 Report of AlexNet

--- slideOptions: transition: slide --- # Pytorch Tainan 090118 Report of AlexNet Reporter : Chen Wei Yu (Graduate student of NCKU ME) --- ## Outline 1. Introduction 2. The Dataset 3. The Architecture 4. Reducing Overfitting 5. Details of learning 6. Result --- ## 1 Introduction + Contributions + Final architecture + Supplement (1) - Overfitting problem V.S. Underfitting problem + Supplement (2) - Fully connected NN V.S. Convolutional NN ---- ### Contribution 1: The authors trained one of the largest convolutional neural networks to date on the subsets of ImageNet used in the ==ILSVRC-2010== and ==ILSVRC-2012== and achieved ==by far the best results ever reported on these datasets==. ---- ### Contributioin 2: They implement the 2D convolution and other operations with the ==high-optimization code on GPU==. (GPUs : GTX 580 3GB $\times\,2$) ---- ### Contribution 3: Their network contains a number of new and unusual features which improve its performance and reduce its training time. Detailed parts will be discussed in the ==Section 3==. ---- ### Contribution 4: Due to the size of their neural network which had ==overfitting problem==, even with the large scale labeled training examples. They used several effective techniques for preventing overfitting, which are described in ==Section 4==. ---- ### Final architecture (1) ![](https://i.imgur.com/y9sAKLW.png) ---- ### Final architecture (2) ![](https://i.imgur.com/y9sAKLW.png) + 5 convolutional layers (grean block) + 3 fully connection layers (red block) ---- ### Supplement (1) #### Overfitting problem V.S. Underfitting problem ![](https://i.imgur.com/yqV8dpn.png) ---- ### Supplement (1) #### Overfitting problem V.S. Underfitting problem + Underfitting problem : High bias(One uses too few parameters to model the problem. ==Even the the prediction of training set and their labels have large difference.==) + Solution : Using a more complicated model. + Overfitting problem : High variance(One uses too much parameters to model the problem. ==The prediction of model is close to the training set but is not general for other testing.==) + Solution : Adding more data, regulations..etc. ---- ### Supplement (1) #### Overfitting problem V.S. Underfitting problem ![](https://i.imgur.com/1pXmDsl.png) ---- ### Supplement (2) #### Fully connected NN V.S. Convolutional NN Neural Networks receive an ==input (a single vector)==, and transform it through a series of ==hidden layers==. Each hidden layer is made up of a set of neurons, where each neuron is ==fully connected== to all neurons in the ==previous layer==, and where neurons in a single layer function completely ==independently== and do not share any connections. The last fully-connected layer is called the ==output layer== and in classification settings it represents the class scores. ---- ### Supplement (2) #### Fully connected NN V.S. Convolutional NN ![](https://i.imgur.com/ouRqWRk.png) ---- ### Supplement (2) #### Fully connected NN V.S. Convolutional NN *3D volumes of neurons*: The layer of a convolutional neural network have neurons arranged in 3 dimensions: ==**width, height, depth.**== ![](https://i.imgur.com/DxLcJJs.png) ---- ### Supplement (2) #### Fully connected NN V.S. Convolutional NN A typically convolutional layer is gernally build with 3 types layers: ==**Convolutional Layer**==, ==**Pooling Layer**==, and ==**Fully-Connected Layer**== (exactly as seen in ++regular Neural Networks++). ---- ### Supplement (2) #### Fully connected NN V.S. Convolutional NN ##### Convolutional layer(1) There are three main properties of convolutioanl layer: ==*Local Connectivity*==, ==*Spatial arrangement*== and ==*Parameter Sharing*==. ---- ### Supplement (2) #### Fully connected NN V.S. Convolutional NN ##### Convolutional layer(2) *Local Connectivity* : We will connect each neuron to only a ==local region of the input volume==. The spatial extent of this connectivity is a hyperparameter called the ==*receptive field*== of the neuron (equivalently this is the ==filter size==). The extent of the connectivity along the depth axis is always equal to the depth of the input volume. ---- ### Supplement (2) #### Fully connected NN V.S. Convolutional NN ##### Convolutional layer(3) ![](https://i.imgur.com/oVPidtu.png) The red part is the ==*receptive field(filter/kernel)*==. ---- ### Supplement (2) #### Fully connected NN V.S. Convolutional NN ##### Convolutional layer(4) *Spatial arrangement* : Three hyperparameters control the size of the output volume: the ==*depth*, *stride* and *zero-padding*==. ---- ### Supplement (2) #### Fully connected NN V.S. Convolutional NN ##### Convolutional layer(5) We can compute ==the spatial size of the output volume== as a function of ==the input volume size ($W$), the receptive field size of the Conv Layer neurons ($F$), the stride with which they are applied ($S$), and the amount of zero padding used ($P$) on the border==. + $(W+2P-F)/S+1$. ---- ### Supplement (2) #### Fully connected NN V.S. Convolutional NN ##### Convolutional layer(5) *Parameter Sharing* : Parameter sharing scheme is used in Convolutional Layers to control the number of parameters. ---- ### Supplement (2) #### Fully connected NN V.S. Convolutional NN ##### Pooling layer(1) Its function is to progressively ==reduce the spatial size== of the representation to ==reduce the amount of parameters and computation== in the network, and hence to also ==control overfitting==. ---- ### Supplement (2) #### Fully connected NN V.S. Convolutional NN ##### Pooling layer(2) ![](https://i.imgur.com/ysZh6ve.png) --- ## 2 The Dataset The authors used the ==ILSVRC-2010 and ILSVRC-2012==. ILSVRC is the acronym of "ImageNet Large-Scale Visual Recognition Challenge". The authors used the ==raw RGB values of the pixels==. The system requires a constant size image. So the author down-sampled the different resolutions images from ImageNet to a fixed resolution of $256 × 256$. ---- ### Supplement (3) #### What is top-1 and top-5 error rate in ILSVRC? + Top-1 : One check if the top class (the one having the highest probability) is the same as the target label. + Top-5 : One check if the target label is one of your top 5 predictions (the 5 ones with the highest probabilities). --- ## 3 The Architecture ![](https://i.imgur.com/LdfAnbc.png) 8 learned layers + 5 convolution layers + 3 fully convolution layers ---- ### 3.1 ReLU Nonlinearity The authors use the ReLU activation function on the system. They found that training with ReLU is faster than using TanH. ![](https://i.imgur.com/yJqHkrK.png) ---- ### Supplement (4) #### Activation function(transfer function) (1) ![](https://i.imgur.com/jlhmofd.png) ---- ### Supplement (4) #### Activation function(transfer function) (2) ![](https://i.imgur.com/maa0ZUF.png) ---- ### 3.2 Training on Multiple GPUs (1) The authors use ==two GPUs== to parallelize the network. Both GPUs can r.w. from another devices ==without accessing host memory==. They puts ==half of kernels(neurons)== on each GPU. And GPUs communicate in certain layers. ---- ### 3.2 Training on Mutiple GPUs (2) ![](https://i.imgur.com/5PnBHRj.png) + *Star* means communication. ---- ### 3.3 Local Response Normalization(1) ReLUs have the desirable property that they ==do not require input normalization== to prevent them from saturating. If at least some training examples produce a positive input to a ReLU, learning will happen in that neuron. However, the authors find that the following local normalization scheme aids generalization. $b^{i}_{x,y}=a^{i}_{x,y}/\begin{pmatrix}k+\alpha \Sigma^{min(N-1,i+n/2)}_{j=max(0,i-n/2)}(a^{i}_{x,y})^{2}\end{pmatrix}^{\beta}$ ---- ### 3.3 Local Response Normalization(2) + $a^{i}_{x,y}$ : the activity of a neuron computed by applying kernel $\mathit{i}$ at position after the ReLU nonlinearity + $b^{i}_{x,y}$ : the response-normalized activity + $N$ : the total number of kernels in the layer + $k$, $n$, $\alpha$ and $\beta$ : hyper-parameters whose values are determined using a validation + $k=2$, $n=5$, $\alpha=10^{-4}$ and $\beta=0.75$ ---- ### 3.4 Overlapping Pooling(1) A pooling layer can be thought of as consisting of ==a grid of pooling units spaced $s$ pixels apart (stride)==, each summarizing ==a neighborhood of size $z × z$== centered at the location of the pooling unit. + If $s=z$, this the gerneral form of pooling layer. + If $s<z$, the authors adapt this technique and find that training with overfitting is more difficult to overfit. $s=2,z=3$ <span>Result : Reducing the top-1 and top-5 error rates by 0.4% and 0.3%</span> ---- ### 3.4 Overlapping Polling(2) ![](https://i.imgur.com/U9GWoJs.png) ---- ### 3.5 Overall Architecture (1) ![](https://i.imgur.com/LdfAnbc.png) ---- ### 3.5 Overall Architecture (2) The output of the last fully-connected layer is fed to a ==1000-way softmax== which produces a distribution over the ==1000 class labels==. Our network maximizes the multinomial logistic regression objective, which is equivalent to maximizing the average across training cases of the log-probability of the correct label under the prediction distribution. + Softmax function : $\sigma(z)_{j}=\frac{e^{z_{j}}}{\Sigma^{K}_{k=1}e^{z_{k}}}$ for $j=1\sim K$ --- ## 4 Reducing Overfitting + Data Augmentation + Dropout ---- ### 4.1 Data Augmentation The easiest and most common method to reduce overfitting on image data is to artificially ==enlarge the dataset using label-preserving transformations.== There are two method used by authors in the system. ---- ### 4.1 Data Augmentation #### $1_{st}$ method The first form of data augmentation consists of generating ==image translations and horizontal reflections.== Without this scheme, their network suffers from substantial overfitting, which would have forced their to use much smaller networks. ---- ### 4.1 Data Augmentation #### Supplement (5) : Common geometric method for data augmentation ![](https://i.imgur.com/2JUGdwD.png) ---- ### 4.1 Data Augmentation #### $2_{nd}$ method The second form of data augmentation consists of ==altering the intensities of the RGB channels in training images.== $\begin{split}I_{xy_{Aug}}&= \begin{bmatrix}I^{R}_{xy_{Aug}},I^{G}_{xy_{Aug}},I^{B}_{xy_{Aug}} \end{bmatrix}^{T}\\ &=\begin{bmatrix}I^{R}_{xy_{Raw}},I^{G}_{xy_{Raw}},I^{B}_{xy_{Raw}} \end{bmatrix}^{T}\\ &+\begin{bmatrix} \mathit{p}_{1_{Raw}}, \mathit{p}_{2_{Raw}},\mathit{p}_{3_{Raw}} \end{bmatrix}\times\\& \begin{bmatrix} \alpha_{1_{Raw}}\lambda_{1_{Raw}},\alpha_{2_{Raw}}\lambda_{2_{Raw}},\alpha_{3_{Raw}}\lambda_{3_{Raw}} \end{bmatrix}^{T} \end{split}$ ---- ### 4.1 Data Augmentation #### Supplement (6) : Mean vector and convariance matrix of pixels (1) + RGB mean vector of image: $\begin{split}\begin{bmatrix} I^{R}_{mean},I^{G}_{mean},I^{B}_{mean} \end{bmatrix} &=\begin{bmatrix} \frac{\Sigma_{i=0}^{N-1}I^{R}}{N},\frac{\Sigma_{i=0}^{N-1}I^{G}}{N},\frac{\Sigma_{i=0}^{N-1}I^{B}}{N} \end{bmatrix} \end{split}$ + RGB covariance matrix of each pixels $I_{xy}$ : $\begin{split}\mathit{Cov}(I_{xy})&=\mathit{E} \begin{pmatrix} \begin{bmatrix} I_{xy}-I_{\,\mathit{mean}} \end{bmatrix} \begin{bmatrix} I_{xy}-I_{\,\mathit{mean}} \end{bmatrix}^{T} \end{pmatrix} \end{split}$ ---- ### 4.1 Data Augmentation #### Supplement (6) : Mean vector and convariance matrix of pixels (2) + $\mathit{p}_{i}$ : $i$th eigenvector of $\mathit{Cov}(I_{xy})$ + $\mathit{\lambda}_{i}$ : $i$th eigenvalue of $\mathit{Cov}(I_{xy})$ + $\alpha_{i}\,\sim\,\mathit{N}(0,0.1)$ : $\alpha_{i}$ is drawn when training the image and it is same for all pixels in the image. It will be respawn when we retrain the same image. ---- ### 4.1 Data Augmentation #### Summary + Why $2_{nd}$ method work? Object identity is invariant to changes in the intensity and color of the illumination. + Result : This scheme reduces the top-1 error rate by over 1%. ---- ### 4.2 Dropout To successfully reduce test errors, one can combine the predictions of many different models. But it takes lots of time to train. The authors use the dropout technique to solve the problem. ---- ### Supplement (7) #### Non-Dropout NN V.S. Dropout NN(1) ![](https://i.imgur.com/tHwLXqg.png) ---- ### Supplement (7) #### Non-Dropout NN V.S. Dropout NN(2) Dropout ==prevents overfitting== and provides a way of ==approximately combining exponentially many different neural network architectures efficiently.== ---- ### Supplement (7) #### Introduction of Dropout NN(1) ==Dropout means that we temporarily remove a neuron. The choice of which units to drop is random.== Each unit is retained with a fixed probability $\mathit{p}$ indepedent of other units. + Input units: the optimal probability $\mathit{p}\sim 1\gg 0.5$ + Non-input units: + Using a validation set + Simplify to set $\mathit{p}=0.5$ ---- ### Supplement (7) #### Introduction of Dropout NN(2) We can think dropout as ==sampling a "thinned" network from original neural nwetwork.== If there are $\mathit{n}$ units in the neural net, then there are $2^{n}$ possible thinned neural networks. The total number of parameters is $\mathit{O}(n^{2})$ due to the network share weights. ---- ### Supplement (7) #### Introduction of Dropout NN (3) ![](https://i.imgur.com/mGkpNIO.png) ---- ### Supplement (7) #### Introduction of Dropout NN (4) ==If a unit is retained with probability $\mathit{p}$ during traning, the outgoing weights of that unit are multipled by $\mathit{p}$ at test time.== This eusures that for any hidden unit the expected output is the same as the actual output at test time. ---- ### Supplement (7) #### Introduction of Dropout NN (5) + $\mathit{l}\in\{1,\cdots,\mathit{L}\}$ : index the hidden layers of the network + $\mathbf{z}^{(\mathit{l})}$ : the vector of inputs into layer $l$ + $\mathbf{y}^{(\mathit{l})}$ : the vector of outputs from layer $l$ + $\mathbf{y}^{0}=\mathbf{x}$ : the input + $W^{(l)}\,and\,\mathbf{b}^{(l)}$ : weights and biases at layer $\mathit{l}$ ---- ### Supplement (7) #### Introduction of Dropout NN (6) ![](https://i.imgur.com/zpsH9Bu.png) ---- ### Supplement (7) #### Introduction of Dropout NN (7) Without dropout, the neural network can be described as (for $\mathit{l}\in\{0,\cdots,\mathit{L}-1\}$ and any hidden unit $\mathit{i}$) $\begin{cases} \mathit{z}^{(\mathit{l}+1)}_{i} = \mathit{w}^{(\mathit{l}+1)}_{\mathit{i}}\mathit{y}^{\mathit{l}}+b^{(\mathit{l})}_{i}\\ \mathit{y}^{(\mathit{l}+1)}_{i} = \mathit{f}\,(\mathit{z_{i}}^{(\mathit{l}+1)}) \end{cases}$ ---- ### Supplement (7) #### Introduction of Dropout NN (8) With dropout, ![](https://i.imgur.com/3kA6tth.png) ---- ### Supplement (7) #### Introduction of Dropout NN (9) With dropout, $\begin{cases} \mathit{r}_{j}^{(\mathit{l})}\,\sim\,Bernoulli(\mathit{p})\\ \tilde{\mathbf{\mathit{y}}}^{(\mathit{l})} = \mathbf{r}^{(l)} \ast \mathit{y}^{(\mathit{l})}\\ \mathit{z}_{i}^{\mathit{(l+1)}} = \mathit{w}_{i}^{(\mathit{l}+1)}\tilde{\mathit{y}}^{\mathit{l}}+b_{i}^{\mathit{(l+1)}}\\ \mathit{y}^{(\mathit{l}+1)}_{i} = \mathit{f}\,(\mathit{z_{i}}^{(\mathit{l}+1)}) \end{cases}$ ---- ### Supplement (7) #### Introduction of Dropout NN (10) The difference between with and without dropout is that ==forward and backpropagation with dropout are only done on thinned network.== The gradients for each parameter are averaged over the training cases in each mini-batch. Any training case which ==does not use a parameter== contributes ==a gradient of zero for that parameter.== --- ## 5 Details of learning The authors use the stochastic gradeient descent on the system. + Momentum updating $\mathit{v}_{i+1} := 0.9v_{i}-0.0005\epsilon\mathit{w}_{i}-\epsilon <\frac{\partial\mathit{L}}{\partial\mathit{w}}|_{\mathit{w_{i}}}>_{D_{i}}$ + Weight updating $\mathit{w}_{i+1} := \mathit{w}_{i}+ \mathit{v}_{i+1}$ ---- ### Supplement (8) : Momentum ![](https://i.imgur.com/6tV56Zn.png) --- ## 6 Result : ILSVRC-2010 ![](https://i.imgur.com/OAnLQfB.png) --- ## 6 Result : ILSVRC-2012 ![](https://i.imgur.com/H545vHt.png) --- ## Thanks for listening !!! --- ## Reference 1. [ImageNet Classification with Deep Convolutional Neural Networks](https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf) 2. [Dropout: A Simple Way to Prevent Neural Networks from Overfitting](http://jmlr.org/papers/volume15/srivastava14a.old/srivastava14a.pdf) 3. [AlexNet 架構概述](https://medium.com/@WhoYoung99/alexnet-%E6%9E%B6%E6%A7%8B%E6%A6%82%E8%BF%B0-988113c06b4b) 4. [AlexNet Introduction](https://slidesplayer.com/slide/11579438/) 5. [【卷积神经网络-进化史】从LeNet到AlexNet](https://zhuanlan.zhihu.com/p/21562756) 6. [CS229 Bias-Variance and Error Analysis](http://cs229.stanford.edu/section/error-analysis.pdf) 7. [李宏毅教授 Machine Learning - Where does the error come from](http://speech.ee.ntu.edu.tw/~tlkagk/courses/ML_2016/Lecture/Bias%20and%20Variance%20(v2).pdf) 8. [CS229 Part VI Learning Theory](http://cs229.stanford.edu/notes/cs229-notes4.pdf) 9. [CS231N Convolutional Neural Network for Visual Recongnition](http://cs231n.github.io/convolutional-networks/#fc) 10. [PyTorch Tainan 聚會 #2 - MLP in NumPy](https://hackmd.io/Xh9OHIgPQuaZ3JCRHSKU0A) 11. [How to implement a neural network](http://peterroelants.github.io/posts/neural_network_implementation_part01/) 12. [ImageNet: what is top-1 and top-5 error rate?](https://stats.stackexchange.com/questions/156471/imagenet-what-is-top-1-and-top-5-error-rate) 13. [Activation function](https://en.wikipedia.org/wiki/Activation_function) 14. [深度學習：使用激勵函數的目的、如何選擇激勵函數 Deep Learning : the role of the activation function](http://mropengate.blogspot.com/2017/02/deep-learning-role-of-activation.html) 15. [李宏毅教授 Machine Learning - Tips for Deep Learning](http://speech.ee.ntu.edu.tw/~tlkagk/courses/ML_2017/Lecture/DNN%20tip.pdf)

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.