---
title: Automatic Differentiation
tags: Docència
slideOptions:
theme: white
transition: 'fade'
---

<style>

.reveal section img { background:none; border:none; box-shadow:none; }
.reveal {
font-size: 30px;
}
.reveal p {
text-align: left;
}
.reveal ul {
display: block;
}
.reveal ol {
display: block;
}

</style>

---

# Automatic Differentiation, Differential programming

Let's consider a simple cost function for a 1-layer (1-D) neural network:

$$ C(x,y,w,b) = y - \max(0,w \cdot x + b)$$

def C(x,y,w,b):
    if w*x+b > 0:
        return y - (w*x+b)
    else:
        return y

Because **automatic differentiation can only calculate the partial derivative of an expression on a certain point**, we have to assign initial values to each of the variables.

Let us say: $x=1$, $w=2$, $y=5$ (this is the expected value), and $b=1$.

The first step is to (**automatically**) convert it into a computational graph:


:::info
:bulb: The `max` operator is implemented with an `if` statement in Python.
:::

Then, we populate the variables with their values (forward path):

| Node | Expression | Value |
| ---- | ---------- | ----- |
| $w_1$ | $w$ | $2$ |
| $w_2$ | $x$ | $1$ |
| $w_3$ | $b$ | $1$ |
| $w_4$ | $y$ | $5$ |
| $w_5$ | $w_1 \cdot w_2$ | $2$ |
| $w_6$ | $w_5 + w_3$ | $3$ |
| $w_7$ | $\max(0,w_6)$ | $3$ |
| $w_8$ | $w_4 - w_7$ | $2$ |
| $z$ | $w_8$ | $2$ |

Next, we need to calculate the partial derivatives of each connection between operations, represented by the edges.


Now we can move on to calculating the partials! Let’s find the partial of with respect to the weights.


Now we simply multiply up the edges:

And that’s our partial!