Neural Networks

This is my personal notes taken for the course Machine learning by Standford. Feel free to check the assignments.
Also, if you want to read my other notes, feel free to check them at my blog.

I) Representation

Neural networks offer an alternate way to perform machine learning when we have complex hypotheses with many features. In other words, Neural Networks have the ability to learn and model non-linear and complex relationships between inputs and outputs.

• Layer 1 is called input layer.
• Layer 3 is called output layer.
• Everything between Layer 1 and Layer 3 are called hidden layer (Layer 2). There can be more than 1 hidden layer.
• Links from Layer 1 to Layer 2 and Layer 2 to Layer 3 are called weights. They are the
  $\theta$s parameters. All
  $\theta$s are gathered in
  ${\mathrm{\Theta }}^{(l)}$, matrix of weights from layer
  $l$ to
  $l+1$.
• $b_0$ is called bias. It is always equal to 1.
• $a_j^{(l)}$ = the
  $j^{th}$ [activation node]{.underline} in layer l.

The input matrix is denoted as:

$$x=\left[\begin{array}{c}{b}_0\\ x_1\\ x_2\\ x_3\end{array}\right]$$

The weight matrix is denoted as:

$${\mathrm{\Theta }}^{(1)}=\left[\begin{array}{cccc}{\theta }_{1,0}^{(1)}& {\theta }_{1,1}^{(1)}& {\theta }_{1,2}^{(1)}& {\theta }_{1,3}^{(1)}\\ {\theta }_{2,0}^{(1)}& {\theta }_{2,1}^{(1)}& {\theta }_{2,2}^{(1)}& {\theta }_{2,3}^{(1)}\\ {\theta }_{3,0}^{(1)}& {\theta }_{3,1}^{(1)}& {\theta }_{3,2}^{(1)}& {\theta }_{3,3}^{(1)}\\ {\theta }_{4,0}^{(1)}& {\theta }_{4,1}^{(1)}& {\theta }_{4,2}^{(1)}& {\theta }_{4,3}^{(1)}\\ {\theta }_{5,0}^{(1)}& {\theta }_{5,1}^{(1)}& {\theta }_{5,2}^{(1)}& {\theta }_{5,3}^{(1)}\end{array}\right]$$

For example:

• ${\theta }_{1,0}^{(1)}$ is the weight from
  $b_0$ to
  $a_1^{(2)}$.
• ${\theta }_{2,2}^{(1)}$ is the weight from
  $x_2$ to
  $a_2^{(2)}$.
• ${\theta }_{5,3}^{(1)}$ is the weight from
  $x_3$ to
  $a_5^{(2)}$.

The indexing could seem weird but it was done this way to prevent from transposing the weight matrix in the hypothesis function.

The activation nodes matrix from the hidden layer is denoted as:

$$a^{(2)}=\left[\begin{array}{c}{a}_1^{(2)}\\ a_2^{(2)}\\ a_3^{(2)}\\ a_4^{(2)}\\ a_5^{(2)}\end{array}\right]$$

II) Forward propagation

To go from one layer to another, we have to do a Forward propagation. It consists on applying an activation function (most of the time the sigmoid function) on a weighted sum. Let's denote g the activation function.

From Layer 1 to Layer 2:

The values for each of the "activation" nodes is obtained as follows:

$$\begin{array}{rl}{a}_1^{(2)}& =g({\theta }_{1,0}^{(1)}{b}_0+{\theta }_{1,1}^{(1)}{x}_1+{\theta }_{1,2}^{(1)}{x}_2+{\theta }_{1,3}^{(1)}{x}_3)\\ a_2^{(2)}& =g({\theta }_{2,0}^{(1)}{b}_0+{\theta }_{2,1}^{(1)}{x}_1+{\theta }_{2,2}^{(1)}{x}_2+{\theta }_{2,3}^{(1)}{x}_3)\\ a_3^{(2)}& =g({\theta }_{3,0}^{(1)}{b}_0+{\theta }_{3,1}^{(1)}{x}_1+{\theta }_{3,2}^{(1)}{x}_2+{\theta }_{3,3}^{(1)}{x}_3)\\ a_4^{(2)}& =g({\theta }_{4,0}^{(1)}{b}_0+{\theta }_{4,1}^{(1)}{x}_1+{\theta }_{4,2}^{(1)}{x}_2+{\theta }_{4,3}^{(1)}{x}_3)\\ a_5^{(2)}& =g({\theta }_{5,0}^{(1)}{b}_0+{\theta }_{5,1}^{(1)}{x}_1+{\theta }_{5,2}^{(1)}{x}_2+{\theta }_{5,3}^{(1)}{x}_3\end{array}$$

which is basically:

$$\begin{gathered} a^{(2)}=g({\mathrm{\Theta }}^{(1)}x) \\ {a}^{(2)}=g(z^{(2)}) \end{gathered}$$

From Layer 2 to Layer 3:

The hidden layer matrix is denoted as:

$$a^{(2)}=\left[\begin{array}{c}{b}_0\\ a_1^{(2)}\\ a_2^{(2)}\\ a_3^{(2)}\\ a_4^{(2)}\\ a_5^{(2)}\end{array}\right]$$

Notice that we have to manually add the bias.

The weight matrix is denoted as:

$${\mathrm{\Theta }}^{(2)}=\left[\begin{array}{cccccc}{\theta }_{1,0}^{(2)}& {\theta }_{1,1}^{(2)}& {\theta }_{1,2}^{(2)}& {\theta }_{1,3}^{(2)}& {\theta }_{1,4}^{(2)}& {\theta }_{1,5}^{(2)}\\ {\theta }_{2,0}^{(2)}& {\theta }_{2,1}^{(2)}& {\theta }_{2,2}^{(2)}& {\theta }_{2,3}^{(2)}& {\theta }_{2,4}^{(2)}& {\theta }_{2,5}^{(2)}\end{array}\right]$$

For example:

• ${\theta }_{1,0}^{(2)}$ is the weight from
  $b_0$ to
  $a_1^{(3)}$.
• ${\theta }_{2,2}^{(2)}$ is the weight from
  $a_2^{(2)}$ to
  $a_2^{(3)}$.
• ${\theta }_{2,5}^{(2)}$ is the weight from
  $a_5^{(2)}$ to
  $a_2^{(3)}$.

The output matrix is denoted as:

$$a^{(3)}=\left[\begin{array}{c}{a}_1^{(3)}\\ a_2^{(3)}\end{array}\right]$$

The values for each of the output nodes is obtained as follows:

$$\begin{array}{rl}{a}_1^{(3)}& =g({\theta }_{0,1}^{(2)}{b}_0+{\theta }_{1,1}^{(2)}{a}_1^{(2)}+{\theta }_{2,1}^{(2)}{a}_2^{(2)}+{\theta }_{3,1}^{(2)}{a}_3^{(2)}+{\theta }_{4,1}^{(2)}{a}_4^{(2)}+{\theta }_{5,1}^{(2)}{a}_5^{(2)})\\ a_2^{(3)}& =g({\theta }_{0,1}^{(2)}{b}_0+{\theta }_{1,1}^{(2)}{a}_1^{(2)}+{\theta }_{2,1}^{(2)}{a}_2^{(2)}+{\theta }_{3,1}^{(2)}{a}_3^{(2)}+{\theta }_{4,1}^{(2)}{a}_4^{(2)}+{\theta }_{5,1}^{(2)}{a}_5^{(2)})\end{array}$$

which is basically:

$$\begin{gathered} a^{(3)}=g({\mathrm{\Theta }}^{(2)}{a}^{(2)}) \\ {a}^{(3)}=g(z^{(3)}) \end{gathered}$$

To go from one layer to another, we have to do a weighted sum (characterized by

$z$) on which we will apply the activation function. That's the principle of the Forward propagation.

III) Intuition

A simple example of applying neural networks is by predicting

$x_1\text{AND}{x}_2$, which is the logical AND operator and is only true if both

$x_1$ and

$x_2$ are 1.

$$\begin{array}{r}\left[\begin{array}{c}{x}_0\\ x_1\\ x_2\end{array}\right]\to \left[\begin{array}{c}g(z^{(2)})\end{array}\right]\to h_{\mathrm{\Theta }}(x)\end{array}$$

Remember that

$x_0$ is our bias variable and is always 1.

Let's set our first theta matrix as:

$${\mathrm{\Theta }}^{(1)}=\left[\begin{array}{ccc}-30& 20& 20\end{array}\right]$$

This will cause the output of our hypothesis to only be positive if both

$x_1$ and

$x_2$ are 1. In other words, we replace the value of

$x_1$ and

$x_2$ by those in the table and then we compare the output of

$h_{\mathrm{\Theta }}(x)$ with the graph of the sigmoid function (above the table).

As we can see, the result is exactly the AND gate. This was possible to do so because we already had the right values of

$\theta$s. Thus, the goal of the neural network is to actually find the right values of

$\theta$s. And to find the right values of

$\theta$s, we have to minimize a cost function. Here is the one used by Andrew Ng: the Cross-Entropy cost function (adapted for Neural Network).

$$\begin{array}{c}C(\mathrm{\Theta })=-\frac{1}{m}\sum _{i=1}^m\sum _{k=1}^K[y_k^{(i)}\log ((h_{\mathrm{\Theta }}(x^{(i)}){)}_k)+(1-y_k^{(i)})\log (1-(h_{\mathrm{\Theta }}(x^{(i)}){)}_k)]+\frac{\lambda }{2m}\sum _{l=1}^{L-1}\sum _{i=1}^{s_l}\sum _{j=1}^{s_{l+1}}({\mathrm{\Theta }}_{j,i}^{(l)}{)}^2\end{array}$$

Remark:

Most of the time, the cost function used in Neural Network are non-convex (it means that it has multiple local minima).

IV) Backward propagation

In order to minimize the cost function, we have to apply the gradient descent.

$${\mathrm{\Theta }}^{(l)}\leftarrow {\mathrm{\Theta }}^{(l)}-\alpha \frac{\partial \mathcal{L}}{\partial {\mathrm{\Theta }}^{(l)}}$$

• ${\mathrm{\Theta }}^{(l)}$ : weights of layer l.
• $\alpha$ : The learning rate.
• $\mathcal{L}$ : Loss function (
  $y\ast \log (\hat{y})+(1-y)\ast \log (1-\hat{y})$)

But how do we compute

$\frac{\partial \mathcal{L}}{\partial {\mathrm{\Theta }}^{(l)}}$ ? By applying the principle of Backpropagation.

Backpropagation is about understanding how changing the weights and biases in a network changes the cost function. Ultimately, this means computing the partial derivatives :

$$\frac{\partial \mathcal{L}}{\partial {\theta }_{jk}^{(l)}}$$

But to compute those, we have to introduce an intermediate quantity

${\delta }_j^l$, which we call the error in the j

${}^{th}$ neuron in the l

${}^{th}$ layer.

There is 2

${\delta }^l$ equations here:

• For the last layer L.
• For hidden layer l.

For the last layer L:

As we can in the picture below, the error in the last layer is cause by the

$a^{(L)}$.

Notice here that

$C_0$ is equivalent to

$\mathcal{L}$. (

$\mathcal{L}$ is the loss function defined on a data point. The cost function

$C$ is the sum of loss functions)

If we differentiate it, we have:

$${\delta }^{(L)}=(a^{(L)}-y)$$

For every other layer l:

To get the delta values of the layers before the last layer, we can use an equation that steps us back from right to left:

$${\delta }^{(l)}=(({\mathrm{\Theta }}^{(l+1)}{)}^T{\delta }^{(l+1)}).\ast a^{\prime }(z^{(l)})$$

The g-prime derivative terms can also be written out as:

$$a^{\prime }(u)=a(u).\ast (1-a(u))$$

The full back propagation equation for the inner nodes is then:

$${\delta }^{(l)}=(({\mathrm{\Theta }}^{(l+1)}{)}^T{\delta }^{(l+1)}).\ast a^{(l)}.\ast (1-a^{(l)})$$

We can then compute our partial derivative terms by multiplying our activation values and our error values for each training example t.

$${\displaystyle \frac{\partial \mathcal{L}}{\partial {\mathrm{\Theta }}_{j,k}^{(l)}}}=\frac{1}{m}\sum _{t=1}^m{a}_j^{(t)(l)}{\delta }_k^{(t)(l+1)}$$

This however ignores regularization, which we'll deal with later.

V) Bonus: Backpropagation

Derivative computation of

${\delta }^{(L)}$:

By applying chain rule,

$${\delta }^{(L)}=\frac{\partial \mathcal{L}}{\partial {\mathrm{a}}^{(\mathrm{L})}}\frac{\partial {\mathrm{a}}^{(\mathrm{L})}}{\partial {\mathrm{z}}^{(\mathrm{L})}}$$

Let's compute

$\frac{\partial \mathcal{L}}{\partial {\mathrm{a}}^{(\mathrm{L})}}$.

We know that:

$$\mathcal{L}(\theta )=-(y_i\ast log(a_i))+(1-y_i)\ast log(1-a_i))$$

where

$$\begin{array}{rl}& a_i=h_{\theta }(x_i)=g(\theta x_i)=\frac{1}{1+e^{-\theta x_i}}\\ & a^{\prime }=a\ast (1-a)\end{array}$$

Thus,

$$\begin{array}{rl}\frac{\partial -(y\ast log(a)+(1-y)\ast log(1-a))}{\partial a}& =-(y\ast \frac{1}{a}+(1-y)\ast -1\ast \frac{1}{1-a})\text{(Chain Rule)}\\ & =(\frac{y}{a}+\frac{(1-y)}{1-a})\end{array}$$

Let's compute

$\frac{\partial {\mathrm{a}}^{(\mathrm{L})}}{\partial {\mathrm{z}}^{(\mathrm{L})}}$.

$\frac{\partial {\mathrm{a}}^{(\mathrm{L})}}{\partial {\mathrm{z}}^{(\mathrm{L})}}$ is just the derivative of the sigmoid function,

$a^L(1-a^L)$

Thus,

$${\delta }^{(L)}=\frac{\partial \mathcal{L}}{\partial {\mathrm{a}}^{(\mathrm{L})}}\frac{\partial {\mathrm{a}}^{(\mathrm{L})}}{\partial {\mathrm{z}}^{(\mathrm{L})}}=(\frac{y}{a}+\frac{(1-y)}{1-a})\ast a^L\ast (1-a^L)=\boxed{{\displaystyle a^L-y}}$$

---

Derivative computation of

${\delta }^{(l)}$

By definition,

$$\begin{array}{rcl}{\delta }_j^l& =& \frac{\partial \mathcal{L}}{\partial z_j^l}\\ & =& \frac{\partial L}{\partial z^{l+1}}\frac{\partial z^{l+1}}{\partial z_j^l}\\ & =& \frac{\partial z^{l+1}}{\partial z_j^l}{\delta }^{l+1}\end{array}$$

We know that,

$$\begin{array}{r}{z}^{l+1}=\sum _j{\mathrm{\Theta }}_j^{l+1}{a}_j^l+b^{l+1}=\sum _j{\mathrm{\Theta }}_j^{l+1}g(z_j^l)+b^{l+1}\end{array}$$

Differentiating, we obtain

$$\begin{array}{r}\frac{\partial z^{l+1}}{\partial z_j^l}={\mathrm{\Theta }}_j^{l+1}{g}^{\prime }(z_j^l)\end{array}$$

Thus,

$$\begin{array}{rc}{\delta }_j^l& ={\mathrm{\Theta }}_j^{l+1}{g}^{\prime }(z_j^l){\delta }^{l+1}\\ {\delta }^l& =\boxed{{\displaystyle ({\mathrm{\Theta }}^{l+1}{)}^T{\delta }^{l+1}{g}^{\prime }(z^l)}}\end{array}$$

---

Computation of

${\displaystyle \frac{\partial \mathcal{L}}{\partial {\mathrm{\Theta }}_{j,k}^{(l)}}}$:

$$\frac{\partial L}{\partial {\mathrm{\Theta }}_{jk}^l}=\frac{\partial \mathcal{L}}{\partial z_j^{l+1}}\frac{\partial z_j^{l+1}}{\partial {\mathrm{\Theta }}_{jk}^l}$$

By definition,

${\delta }_j^{l+1}=\frac{\partial \mathcal{L}}{\partial z_j^{l+1}}$ and since

$z_j^{l+1}=\sum _k{\mathrm{\Theta }}_{jk}^l{a}_k^l+b_j^l$, we have

$\frac{\partial z_j^{l+1}}{\partial {\mathrm{\Theta }}_{jk}^l}=a_k^l$.

And so we have,

$$\frac{\partial \mathcal{L}}{\partial {\mathrm{\Theta }}_{jk}^l}=a_k^l{\delta }_j^{l+1}\text{(Vectorized form)}$$

VI) Random initialization of weights

For example, let's say you initialize all your weights to zero, then all of the hidden neurons (units) in your neural network will be doing the exact same calculations. The goal of Random initialization is to break the symmetry (symmetry breaking). You want different hidden units to compute different functions.

Analogy: Imagine that someone has dropped you and your friends from a helicopter to an unknown mountain at the top and you're trapped there. Everywhere is fogged. The only thing you know is that you should get down to the sea level somehow. Which direction should you take to get down to the lowest possible point?

If you couldn't find a way to the sea level and so, the helicopter would take you guys again and would drop you to the same mountain at the top position. You would have to take the same directions again because you're "initializing" yourself to the same starting positions.

However, each time the helicopter drops you somewhere random on the mountain, you guys would take different directions and steps. So, there would be a better chance for you to reach to the lowest possible point.

This is what is meant by breaking the symmetry. The initialization is asymmetric so that you can find different solutions to the same problem.

In this analogy, where you land is the weights. So, with different weights, there's a better chance of reaching to the lowest (or lower) point.