# Computational Graph
Computational Graph is a language or expression used to describe the process of calculation. It is useful in helping others understand the entire computation. Due to the complexity of deep learning models, it is necessary to introduce Computational Graphs.

e.g.
$$ f(x_1,x_2) =x_1x_2w_0 =u \\ g(u) = w_1u+w_2=z \\ h(z)=5*z=y$$

can be describe as follow Computational Graphs

graph LR
    x[x1,x2]--f-->u[u]--g-->z[z]--h-->y[y]

Square presents variable and Line presents operation

# A sample ANN graph
Assume an ann model is defined as follow graph. where $u_i = \sum_{i\in I} {x_i w_i}$ loss function is defined as $L(\hat{y},y)$.

graph LR
    x1[x1]--w1-->node1([u1])
    x2[x2]--w2-->node1
    x1[x1]--w3-->node2([u2])
    x2[x2]--w4-->node2
    node1--w5-->node3([hat_y])
    node2--w6-->node3

## Target: fit $\hat{Y}$ to $Y$
To fit $\hat{Y}$ and $Y$. In other words, it means minimizing the loss function $L(\hat{y},y)$. Therefore, we need to calculate the derivative with respect to every weight in the model. Then, update the weight by **gradient descent**.
>Target
>$$\frac{\partial L}{\partial w_i}$$
>where $i\;\in I$

## Backward proagation
Beacuse $\frac{\partial L}{\partial w_i}$ can't be calculated directly. We can use backward proagation. It split the $\frac{\partial L}{\partial w_i}$to $\frac{\partial L}{\partial u_j}\frac{\partial u_j}{\partial w_i}$

### Forward pass
First, we can easily calculate weight gradient $u_1\rightarrow u_3$ in small part of model.

$$\frac{\partial u_3}{\partial w_5}$$

graph LR
    x1[x1]--w1-->node1([u1])
    x2[x2]--w2-->node1
    x1[x1]--w3-->node2([u2])
    x2[x2]--w4-->node2
    node1--w5-->node3([u3])
    node2--w6-->node3-->y((y))
style node1 fill:#f9f,stroke:#333,stroke-width:4px
style node3 fill:#f9f,stroke:#333,stroke-width:4px
style y fill:#f9f,stroke:#333,stroke-width:4px

### Backward pass
Second, calculate node gradient $u_3\rightarrow y$ which is related to loss function $L(\hat{y},y)$
$$\frac{\partial L}{\partial u3} =\frac{\partial y}{\partial u_3}\frac{\partial L}{\partial y} $$

graph LR
    x1[x1]--w1-->node1([u1])
    x2[x2]--w2-->node1
    x1[x1]--w3-->node2([u2])
    x2[x2]--w4-->node2
    node1--w5-->node3([u3])
    node2--w6-->node3-->y((Y))


style node3 fill:#f9f,stroke:#333,stroke-width:4px
style y fill:#f9f,stroke:#333,stroke-width:4px

### summary

$$\frac{\partial L}{\partial w5} =\frac{\partial L}{\partial u3}\frac{\partial u3}{\partial w5}$$