---
# System prepended metadata

title: Automatic Differentiation
tags: [Docència]

---

---
title: Automatic Differentiation
tags: Docència
slideOptions:
  theme: white
  transition: 'fade'
---

<style>

.reveal section img { background:none; border:none; box-shadow:none; }
.reveal {
  font-size: 30px;
}
.reveal p {
    text-align: left;
  }
.reveal ul {
    display: block;
  }
.reveal ol {
   display: block;
  }

</style>

---

# Automatic Differentiation, Differential programming

Let's consider a simple cost function for a 1-layer (1-D) neural network:

$$ C(x,y,w,b) = y - \max(0,w \cdot x + b)$$

```python=
def C(x,y,w,b):
    if w*x+b > 0:
        return y - (w*x+b)
    else:
        return y
```

Because **automatic differentiation can only calculate the partial derivative of an expression on a certain point**, we have to assign initial values to each of the variables. 

Let us say: $x=1$, $w=2$, $y=5$ (this is the expected value), and $b=1$.  

The first step is to (**automatically**) convert it into a computational graph:

![](https://i.imgur.com/ALX9QEn.png)

:::info
:bulb: The `max` operator is implemented with an `if` statement in Python.
:::


Then, we populate the variables with their values (forward path):




| Node | Expression | Value |
| ---- | ---------- | ----- |
| $w_1$   | $w$ | $2$ |
| $w_2$   | $x$ | $1$ |
| $w_3$   | $b$ | $1$ |
| $w_4$   | $y$ | $5$ |
| $w_5$   | $w_1 \cdot w_2$ | $2$ |
| $w_6$   | $w_5 + w_3$ | $3$ |
| $w_7$   | $\max(0,w_6)$ | $3$ |
| $w_8$   | $w_4 - w_7$ | $2$ |
| $z$   | $w_8$ | $2$ |

Next, we need to calculate the partial derivatives of each connection between operations, represented by the edges. 

![](https://i.imgur.com/GU9zBaq.png)


Now we can move on to calculating the partials! Let’s find the partial of with respect to the weights.

![](https://i.imgur.com/jZsTT5Q.png)

Now we simply multiply up the edges:


And that’s our partial!

![](https://i.imgur.com/TZYBKDi.png)



