{%hackmd SybccZ6XD %}
###### tags: `paper`

# Rectifier Nonlinearities Improve Neural Network Acoustic Models

## Rectifier Nonlinearities

Problem: sigmoidal DNNs can suffer from the **vanishing gradient problem**
> When the input is a large number, the output is saturation(1 or -1).

Problem: ReLU avoid the saturation when input a positive number
> The gradient is 0 whenever the input is negative.

Why ReLU let the negative inputs equal to zero?
> That is kind of denoise.

Solve: the saturation of ReLU problem.
> leaky ReLU

The comparison between activation function.
> 

The formula of leaky ReLU

## Result

Speech recognition task
> 

Empirical activation probability
> The activation probability of last hidden layer in the 10,000 input samples.
>