# CS231n L4 Introduction to Neural Network
## Introduction

Simpler functions stack on top of each other in a hierarchical way in order to make a complex non-linear function.


:::success
Allow to weight together multiple templates to get the overall final score for the class
:::


### Architecture
*Hidden layer: vector*

## Convolutional Neural Networks
### History
Cat Experiment
Hierarchical organization: different types of cells responding to different visual stimulus

Neurocognitron
sandwich architecture (SCSCSC...)
simple cells(S): modifiable parameters
complex cells(C): perfom pooling

Gradient-based learning
Applied to document recognition using backpropagation

Deep Convolutional Neural Networks
ImageNet Classification

### Introduction

Calculate the dot product of the image and the filter (plot the filter on different spots on the image).

ConvNet: Sequence of Conv layers, interspersed with activation functions.

Output of the previous conv layer = Input of the next conv layer

Filters on earlier layer: low-level features (like results in cat experiment)

Increasing depths: increasing the number of filters (works well irl)

:::success
Preserve spatial structure
:::

### Conv Layer (Activation Map)
E.g. Stride of 1 (3x3 filter)
(stride of 3 won't work)

N = width(length), F = filter size

*In practice: Padding zeros around the border (maintain full size output -> won't shrink too fast)*

:::success
An activation map is a big sheet of neuron outputs. Each is connected to a small region in the input.
:::

#### Brain View of Conv Layer
How much this neuron has been triggered at every spatial location in the image.

Preserve spatial structure, reasoning on top of activation maps.

### Pooling Layer (Downsampling)
Makes the representations smaller and more manageable.

Operates over each activation map independently.

Doesn't change the input depth.

Won't use zero-padding during pooling.

#### Max Pooling
Taking the maximum value in each filter.

Why max pooling?
Meaning of value: how much this filter fired the region.

### Activation Functions
#### Sigmoid
Ranging from 0 to 1, interpreted as **firing rate** of a neuron.
:::danger
Problems:
* Saturated neurons **kill** the gradients (e.g. x = -10, x = 10...). Can't pass down gradient flow (gradient = 0).
* Sigmoid outputs are not zero-centered. Inefficient gradient updates. (only allow updates in same direction)
* exp() is compute expensive
:::

#### tanh(x)
Ranging from -1 to 1. **Zero centered.** But still kills gradients when saturated.


#### ReLU

* Doesn't saturated (in pos region).
* Converges faster than sigmod/tanh.
* More biologically plausible. (Similar inputs/outputs of real neurons).
* **Not zero-centered**


:::danger
Problem:

Learning rate too big -> Function well in the beginning but won't work later on.
:::

#### Leaky ReLU
* Doesn't saturated
* Fast and efficient
* Doesn't die


#### PReLU
More flexibility, using alpha instead of hard-coded number (0.01).


#### ELU
Exponential Linear Units
* All benefits of ReLU
* Closer to zero mean outputs
* **More robustness** to noise compared to Leaky ReLU
* **Computation requires exp()**


#### Maxout Neuron
Generalizes ReLU and Leaky ReLU, taking the maximum of two functions.

This doubles the number of parameters per neuron.

### Data Preprocessing

*Not commonly used in processing images*

:::success
For images, center only. (Don't want to downgrade the dimension).
:::

### Weight Initialization
#### Random Assignment
For small networks, randomly generating numbers for weights is okay.

Random assignment won't work for deep networks. If weights too small, all activations = 0. If weights too big, gradients = 0 (saturated).

*Activations = 0*

*Gradients = 0*

#### Xavier Initialization
Many inputs -> smaller weights.


#### For ReLU
Because it's killing half of the units (setting them to 0), has to add additional **/2** at the end.

### Moniting Training and Adjusting Parameters
1. Preprocess the data
2. Choose the architecture (e.g. 1 hidden layer of 50 neurons)
---Start of sanity check---
3. Check if the loss is reasonable
4. After adding regularization, loss should go up
5. Try to get small loss on small dataset
---End of sanity check---
6. Find the learning rate that makes loss go down

### Hyperparameter Optimization
Grid Search v.s. Random Search


*Grid: missing important values*

*Changing learning rate*

*Big gap between training / validation accuracy -> overfitting*

### Optimization
Problems with SGD:
* Too many directions in high dimensional problem -> zigzagging
* Loss function has a local minima or saddle point (gradient = 0)

#### First-Order Optimization

##### SGD + Momentum
*Build up **velocity** as a running mean of gradients.*


*Velocity building up while moving downhill, passing through saddle points*


##### AdaGrad
*Add **grad_squared** term (historical sum of squares in each dimension) while updating gradients. To decrease the phenomenon of zigzagging.*

:::success
Step size getting smaller over time. (Might get stuck at saddle points)
:::
To solve this problem:

##### Adam
*RMSProp with momentum.*


:::info
Learning rate decay over time during training is commonly used.

:::

#### Second-Order Optimization

Stepping toward the minimum of the quadratic -> Doesn't require learning rate.

E.g. L-BFGS

### Prevent Overfitting
#### Model Ensembles
Train multiple independent models or average their results at test time.

Increase performance by several percentage.

Polyak averaging: Instead of using actual parameter vector, keep a moving average of the parameter vector and use that at test time.

#### Regularization
*Improve single-model performance.*

Training: Add some randomness
Testing: Average out randomness

##### Dropout
Randomly set some neurons to 0 in each forward pass.

Probability of dropping (hyperparameter) is usually set to 0.5.

Left: original version ; Right: dropped out version

At test time, multiply by dropout probability => approximate the integral

Inverted dropout: do the division at training time to prevent multiplication at test time

##### Batch Normalization

*To make gaussian activations. To increase efficiency (allow higher learning rate) and prevent gradients becoming 0.*

Inserted after Fully Connected or Convolutional layers, and before nonlinearity(e.g. tanh).

*Retrieved from https://medium.com/ching-i/batch-normalization-%E4%BB%8B%E7%B4%B9-135a24928f12*

*Normalizing minibatch (inputs)*

*Allow the network to squash the range*

*Scaling and shifting*

##### Data Augmentation
Adding random transformations of the input data
* Image crop / flip / rotation
* Color jitter (randomize contrast and brightness)

##### Drop Connect
Rather then zeroing out activations at every forward path, randomly zero out some of the values of the weight matrices.

##### Fractional Max Pooling
Randomize the pooling region.

##### Stochastic Depth
Randomly drop layers during training. But use the whole network during test time.

#### Transfer Learning
Doesn't require huge dataset to train CNN.

For bigger dataset, lower learning rate when finetuning (e.g. 1/10).