問答題

What is softmax used for?

What are three supervised cnn model usage?

What are the three major issues you need to learn when you study a neural network model?

• Network Architecture
• Activation function
• Learning rule

Basic CNN architecture can be divided into two stages, what are these stages? What are the functions of the corresponding two stages?

• Convolutional layers+ pooling layer
  • Feature extraction(特徵擷取)
• Fully connected layer
  • Mapping of feature maps to target labels(分類)

計算題

TLU

原版

• Activation =
  $a=w_1{x}_1+w_2{x}_2+w_3{x}_3....$
• Y=1 if
  $a\ge \theta$, else y=0
• Update weight =
  $w^{\prime }=w+a(t-y)V$
  • $a$ is the learning rate, V is the input vector for example
    $x_1=1,x_2=0$

TLU example

• 2 input or gate(if one of the input is true )
• Initial Weight
  $w_1=1,w_2=2,\theta =2$
• Time = 1
  • $0\times 1+0\times 2=0$
  • $A<\theta \to y=0$
  • $y=0\to$
  • $t-y=0$ no change
• Time = 2
  • no change
• Time =3
  • $1\times 1+0\times 2=1$
  • $A<\theta \to y=0$
  • $w_1^{\prime }=1+0.5(1-0)\times 1=1.5$
  • $w_2^{\prime }=2+0.5(1-0)\times 0=2$
• … Continue until weight fits all condition

Answer

• No, because Activation condition is reversed
• change learning rule to
  $w^{\prime }=w-a(t-y)x$

Perceptron

a

• 2 layer, 2 input 2 output percetron

b

• No , not linearly seperable by one line

c

• $\mathrm{\Sigma }{s}_1=0.2\ast 1+(-0.5)\ast 1=-0.3$
• $\mathrm{\Sigma }{s}_2=(-0.7)\ast 1+0.5\ast 1=-0.2$
• $w_{1,1}^{\prime }=w_{1,1}+0.5(0-0)\ast 1=0.2$
• $w_{1,2}^{\prime }=w_{1,2}+0.5(0-0)\ast 1=-0.5$
• $w_{2,1}^{\prime }=w_{2,1}+0.5(1-0)\ast 1=-0.2$
• $w_{2,2}^{\prime }=w_{2,2}+0.5(1-0)\ast 1=1$

*　

*　Neural Networks and Machine Learning, Simon Haykin, 3nd ed., Pearson, 2009 p85(pdf)
*　

$y(n)\mathrm{為}\mathrm{經}\mathrm{過}activation\mathrm{的}\mathrm{輸}\mathrm{出}$

Back propogation

a

• Input layer 2, hidden 2, output layer 2

b

• Note
  • $w$ is the weight
  • $a$ is the result of activation on the node
    $a=f(\mathrm{\Sigma }(w_{ji}{a}_i))$
  • $\eta$ is the learning rate
  • ${\delta }_j=(d_j-a_j)\ast f^{\prime }(s_j)=(d_j-a_j)a_j(1-a_j)$
  • $\mathrm{\Delta }{w}_{ji}=\eta {\delta }_j{a}_i$

$w_{31}$	0.1	$w_{32}$	0.2
$w_{41}$	0.3	$w_{42}$	0.4
$w_{53}$	0.5	$w_{54}$	0.6
$w_{63}$	0.7	$w_{64}$	0.8

• input =
  $(1,1)$

Forward

• $a_3=f(1\ast 0.1+1\ast 0.2)=0.57$
• $a_4=f(1\ast 0.3+1\ast 0.4)=0.67$
• $a_5=f(0.57\ast 0.5+0.67\ast 0.6)=0.67$
• $a_6=f(0.57\ast 0.7+0.67\ast 0.8)=0.72$

$\delta$s

• target (0,1)
• ${\delta }_6=(d_6-a_6)a_6(1-a_6)=(1-0.72)0.72(1-0.72)=0.0564$
• ${\delta }_5=(0-0.67)0.67(1-0.67)=-0.1481$
• ${\delta }_4=({\delta }_5{w}_{54}+{\delta }_6{w}_{64})\ast f^{\prime }(s_4)=(-0.1481\ast 0.6+0.0564\ast 0.8)0.67(1-0.67)=-0.0097$
• ${\delta }_3=({\delta }_5{w}_{53}+{\delta }_6{w}_{63})\ast f^{\prime }(s_3)=(-0.1481\ast 0.5+0.0564\ast 0.7)0.57(1-0.57)=-0.0085$

$\mathrm{\Delta }w$

$\mathrm{\Delta }{W}_{64}=\eta {\delta }_6{a}_4=0.5\ast 0.0564\ast 0.67=0.01889$

$\mathrm{\Delta }{W}_{63}=0.5\ast 0.0564\ast 0.57=0.0161$

$\mathrm{\Delta }{W}_{54}=0.5\ast -0.1481\ast 0.67=-0.0496$

$\mathrm{\Delta }{W}_{53}=0.5\ast -0.1481\ast 0.57=-0.0422$

$\mathrm{\Delta }{W}_{42}=0.5\ast -0.0097\ast 1=-0.0049$

$\mathrm{\Delta }{W}_{41}=0.5\ast -0.0097\ast 1=-0.0049$

$\mathrm{\Delta }{W}_{32}=0.5\ast -0.0085\ast 1=-0.0043$

$\mathrm{\Delta }{W}_{31}=0.5\ast -0.0085\ast 1=-0.0043$

update weight

$w^{\prime }=w+\mathrm{\Delta }w$

$w_{31}^{\prime }$	0.0957	$w_{32}^{\prime }$	0.1957
$w_{41}^{\prime }$	0.2951	$w_{42}^{\prime }$	0.3951
$w_{53}^{\prime }$	0.4578	$w_{54}^{\prime }$	0.5504
$w_{63}^{\prime }$	0.7161	$w_{64}^{\prime }$	0.8189

驗證 tensorflow

https://colab.research.google.com/drive/1uWPPby020fEdusBBwIwjyLRyArkqoEMv?usp=sharing

CNN

Architecture

A forward

• Tensorflow check https://github.com/JINSCOTT/MLHomework/blob/0cd3071fed50c104864d2fe2b8d8b4fbad5122ed/cnn_maxpooling_backpropagate.ipynb

CNN

Pooling

Linear

• $a_5=f(0.1\ast 80+(-0.05)\ast 90+0.05\ast 20+(-0.02)\ast 60)=0.9644$
• $a_6=f(0.05\ast 80+(-0.02)\ast 90+0.03\ast 20+(-0.07)\ast 60)=0.1978$
• $a_7=f(-0.4\ast 0.9644+-1\ast 0.1978)=0.3581$
• $a_8=f(0.5\ast 0.9644+-0.5\ast 0.1978)=0.5947$

Softmax

• $s1=0.4411$
• $s2=0.5589$

Backward

*　因為有softmax層　outputlayer算法為:
*

${\delta }_j=(t_j-y_j)(y_j-y_j^2)a_j(1-a_j)$
*

$t_j$為target，

$y_j$為softmax的結果
*　其他不變

sigmoid

$\delta$s

• target (１，０)
• ${\delta }_8=(0-0.5589)(0.5589-{0.5589}^2)0.5947(1-0.5947)=-0.0332$
• ${\delta }_{７}=(1-0.4411)(0.4411-{0.4411}^2)0.3581(1-0.3581)=0.0317$
• ${\delta }_{６}=({\delta }_8{w}_{86}+{\delta }_7{w}_{76})\ast f^{\prime }(a_6)=(-0.0332\ast -0.5+0.0317\ast -1)0.1978(1-0.1978)=-0.0024$
• ${\delta }_{５}=({\delta }_8{w}_{85}+{\delta }_7{w}_{75})\ast f^{\prime }(a_5)=(-0.0332\ast 0.5+0.0317\ast -0.4)0.9644(1-0.9644)=-0.001$
• ${\delta }_4=({\delta }_6{w}_{64}+{\delta }_5{w}_{54})\ast f^{\prime }(a_4)=(-0.0024\ast -0.07-0.001\ast -0.02)\ast 1=0.000188$
• ${\delta }_3=({\delta }_6{w}_{63}+{\delta }_5{w}_{53})\ast f^{\prime }(a_3)=(-0.0024\ast 0.03-0.001\ast 0.05)\ast 1=-0.000122$
• ${\delta }_2=({\delta }_6{w}_{62}+{\delta }_5{w}_{52})\ast f^{\prime }(a_2)=(-0.0024\ast -0.02-0.001\ast -0.05)\ast 1=0.0001$
• ${\delta }_1=({\delta }_6{w}_{61}+{\delta }_5{w}_{51})\ast f^{\prime }(a_1)=(-0.0024\ast 0.05-0.001\ast 0.1)\ast 1=-0.00022$

sigmoid

$ｗ＋\mathrm{\Delta }w$

• learning rate = 0.5
• $w_{8,6}=-0.5+0.5\ast -0.0332\ast 0.1978=-0.5033$
• $w_{8,5}=0.5+0.5\ast -0.0332\ast 0.9644=0.4839$
• $w_{7,6}=-1+0.5\ast 0.0317\ast 0.1978=-0.9968$
• $w_{7,5}=-0.4+0.5\ast 0.0317\ast 0.9644=0.3847$
• $w_{6,4}=-0.07+0.5\ast -0.0024\ast 60=-0.142$
• $w_{6,3}=0.03+0.5\ast -0.0024\ast 20=0.006$
• $w_{6,2}=-0.02+0.5\ast -0.0024\ast 90=-0.128$
• $w_{6,1}=0.05+0.5\ast -0.0024\ast 80=-0.046$
• $w_{5,4}=-0.02+0.5\ast -0.001\ast 60=-0.05$
• $w_{5,3}=0.05+0.5\ast -0.001\ast 20=0.04$
• $w_{5,2}=-0.05+0.5\ast -0.001\ast 90=-0.095$
• $w_{5,1}=0.1+0.5\ast -0.001\ast 80=0.06$

upsampling

• reverse maxpool
  
  
• reverse relu
  
  
  
  
  
  

IOU Calculate

chapter 6 Object detection

Difference between one stage and two stage

R-CNN

• 重複製作對象feature map造成速度較慢
• Hard to optimize
• Have to be trained seperately

yolo family

• single shot detector

Yolov1

• backbone: based on GoogLeNet
• Unified detection:
  • 
  • Non-Maximum Suppression (NMS) 用來選去包圍物件最佳的　Bounding box.獲得最佳的 Intersection over Union (IoU)
  • 
• 優點：　快、　訓練較簡單
• 缺點：
  • 一個Grid只能有一(或2)個Class，因此對於擁擠及較小的物間偵測能力較差
  • bounding box 對於物件的aspect ratio 較為固定

Yolov2

• 更換 backbone 為 Darknet 19
• Remove the fully connected layers with average pooling
• More BBOX(5) in each grid cell(better at small and occlusion)
• multi - scale training on each batch

Yolov3

• Darknet 53
• Residual learning(input is combined with block output)
• 

Yolov4

• CSPDarknet53
• Neck: SPP(splatial pyramid) and PANet(, Path Aggregation Network)
• Head: YOLO layer
• Bounding box regression loss
• CIoU, GIoU, DIoU, MSE 四種
  • IOU Loss
  • CIoU　(Complete-IoU) Loss
• Regularization
• Data augmentation
  • Cut mix: 在圖中放另一張圖
  • Mosaic data augmentation: 將四張圖組合唯一

Yolov5

• More mosaic data augment
• GIoU (Generalized-IoU) Loss
• PANet only
• Implement in pytorch

YOLOX

• Anchor free
  • Faster training/inference speed
  • Do not need to determine anchor parameters
    
    
• Decoupled head
• 

Else

• Yolo F/R/S/P

Chapter 7 Instance segmentation

R-CNN

Fast R-CNN

• Still Use Selective search

Faster R-CNN

• Speed up with region proposal network
  
  

Mask R-CNN

• Extends Faster R-CNN with segmentation
  
  

SOLO family

SOLO

SOLOV2

difference fromv1

• The object mask generation is decoupled into a mask kernel prediction and mask feature learning, which are responsible for generating convolution kernels and the feature maps to be convolved with, respectively.
• Predict high-resolution object masks
• SOLOv2 significantly reduces inference overhead with matrix non-maximum suppression (NMS) technique.
  
  
• Dynamic Convolutions
  • 
  • More flexible
  • It adds 2D offsets to the regular grid sampling locations in the standard convolution. It enables free form deformation of the sampling grid

outdated (no un-supervised)

List three most popular types of generative models.

• Variational Autoencoders
• Pixel RNN/CNN
• Generative adversarial network(GAN)

What is the main difference between Supervised Learning and Un-supervised Learning networks?

• Supervised learning
  • Data: Data and label
  • Goal: Map input data to label
• Un-supervised learning
  • Data: Data, no labels
  • Goal: Learn some underlying structure of the data

Competitive network

SOFM

Cross entropy and Gradient Descent Method

tags: Artificial Neural Networks and Deep Learning CSnote Artificial Neural Networks and Deep Learning