---
title: Artificial Neural Network,ANN
tags: 111course
description: Use `{%hackmd theme-dark %}` syntax to include this theme.
---
# ***<div id="animation_title">Artificial Neural Network</div>***
<font font-size:29px>Edited by $\mathfrak{Warren}$ </font>
:::spoiler 文章目錄
[toc]
:::
## What is <span class="sblue">Neural</span> <span class="cblue">Network</span>?
### *<span class="sblue">Neural</span>*: what are the neurons?
+ Neuron<span class="kblue">(神經元)</span>$\rightarrow$thing that holds a number(activation).
+ Activation<span class="kblue">(激勵值)</span>$\rightarrow$every activaton in neuron in layers would effect the next layer.

### *<span class="cblue">Network</span>*: how are they connected?
+ layer(層)$\rightarrow$there are many neurons in a layer and layers conneted each other.
+ <font color="#fc9ae5">input layer</font>$\longrightarrow$<font color="#9aa9fc">hidden layer1</font>$\longrightarrow$<font color="#9aa9fc">hidden layer2</font>$\longrightarrow$...$\longrightarrow$<font color="#9afcd3">output layer</font>
+ But, why layers?$\rightarrow$simliar to *"devide and conquer"*
+$$
### What are the synapse doing?
+ synapes $\rightarrow$the weight <span class="kblue">(權重值)</span>
+ we calculate the next neuron's activation by
$$
w_1a_1+w_2a_2+w_3a_3+w_4a_4+...+w_na_n
$$
+ But the number we cauculated may be any number
+ **SO**, we apply the **activation function**<span class="kblue">(激勵函數)</span>
+ **activation function**, take an example, ***Sigmoid Function***
$$
\sigma(t)= \frac{1}{1+e^{-t}}
$$
+ Then we put the number we got into the function and would get
$$
\sigma(w_1a_1+w_2a_2+w_3a_3+w_4a_4+...+w_na_n)
$$
+ And the next step, we adjust this number by adding a barrier for it.
+ The barrier is called ***Bias*** which can let we choose the activation which is higher than others.
+ And that would be
$$
\sigma(w_1a_1+w_2a_2+w_3a_3+w_4a_4+...+w_na_n-b)
$$
+ To see the overall, we can write every neuron as a equation.
$$
\sigma\{\left[\begin{array}{cc}
w_{11} & w_{12} & w_{13} & \cdots&w_{1n}\\
w_{21} & w_{22} & w_{23} & \cdots&w_{2n}\\
w_{31} & w_{32} & w_{33} & \cdots&w_{3n}\\
\vdots & \vdots & \vdots & {}^\cdot\cdot_\cdot&\vdots\\
w_{k1} & w_{k2} & w_{k3} & \cdots&w_{kn}
\end{array}\right]
\left[\begin{array}{cc}
a_{1} \\
a_{2} \\
a_{3} \\
\vdots \\
a_{k}
\end{array}\right]+
\left[\begin{array}{cc}
b_{1} \\
b_{2} \\
b_{3} \\
\vdots \\
b_{k}
\end{array}\right]\}
$$
+ Above equation can be written as :
$$
\sigma(W\overrightarrow{a}+\overrightarrow{b})
$$
## **Recent optimizers**
### Gradient descent, GD(梯度下降法)
<span class="sblue">*中文*</span>
[梯度下降法](http://surl.li/dxdoi)
[梯度最佳解相關算法](http://surl.li/dxdom)
[SGD, Momentum, AdaGrad, Adam Optimizer](https://reurl.cc/286exX)
<span class="sblue">*English*</span>
[An overview of gradient descent optimization algorithms](https://ruder.io/optimizing-gradient-descent/index.html#adam)
> 在神經網路中，不論是哪種網路，最後都是在找層和層之間的關係
> (參數，也就是層和層之間的權重)，而找參數的過程就稱為學習，
> 所以神經網路的目的就是不斷的更新參數，去最小化損失函數的值，
> 然後找到最佳解。
### Stochastic gradient descent, SGD(隨機梯度下降法)
### Momentum
### Adagrad
### RMSProp
### Adam

<style>
@-webkit-keyframes A
{
0% { color:#fc4444;
padding-left:0px}
5% { color:;
padding-left:5px;}
10% { color:#fc7844;
padding-left:10px}
15% { color:;
padding-left:15px;}
20% { color:#fc7844;
padding-left:20px}
25% { color:;
padding-left:25px;}
30% { color:#e7fc44;
padding-left:30px}
35% { color:;
padding-left:35px;}
40% { color:#e7fc44;
padding-left:40px}
45% { color:;
padding-left:45px;}
50% { color:#7bfc44;
padding-left:50px}
55% { color:;
padding-left:45px;}
60% { color:#44fc9a;
padding-left:40px}
65% { color:;
padding-left:35px;}
70% { color:#44fc9a;
padding-left:30px}
75% { color:;
padding-left:25px;}
80% { color: #44fc9a;
padding-left:20px}
85% { color:;
padding-left:15x;}
90% { color: #9144fc;
padding-left:10px}
95% { color:;
padding-left:5px;}
100% { color: #fc44d7;
padding-left:0px}
}
#animation_title{
animation: A 3s ease 0s infinite alternate;
-webkit-animation: A 3s ease 0s infinite alternate;

}

html, body, .ui-content {
background-color: #333;
color: #ccc;
}

.markdown-body h1,
.markdown-body h2,
.markdown-body h3,
.markdown-body h4,
.markdown-body h5,
.markdown-body h6 {
color: #ddd;
}

.markdown-body h1,
.markdown-body h2 {
border-bottom-color: #ffffff69;
}

.markdown-body h1 .octicon-link,
.markdown-body h2 .octicon-link,
.markdown-body h3 .octicon-link,
.markdown-body h4 .octicon-link,
.markdown-body h5 .octicon-link,
.markdown-body h6 .octicon-link {
color: #fff;
}

.markdown-body img {
background-color: transparent;
}

.ui-toc-dropdown .nav>.active:focus>a, .ui-toc-dropdown .nav>.active:hover>a, .ui-toc-dropdown .nav>.active>a {
color: white;
border-left: 2px solid white;
}

.expand-toggle:hover,
.expand-toggle:focus,
.back-to-top:hover,
.back-to-top:focus,
.go-to-bottom:hover,
.go-to-bottom:focus {
color: white;
}

.ui-toc-dropdown {
background-color: #333;
}

.ui-toc-label.btn {
background-color: #191919;
color: white;
}

.ui-toc-dropdown .nav>li>a:focus,
.ui-toc-dropdown .nav>li>a:hover {
color: white;
border-left: 1px solid white;
}

.markdown-body blockquote {
color: #bcbcbc;
}

.markdown-body table tr {
background-color: #5f5f5f;
}

.markdown-body table tr:nth-child(2n) {
background-color: #4f4f4f;
}

.markdown-body code,
.markdown-body tt {
color: #eee;
background-color: rgba(230, 230, 230, 0.36);
}

a,
.open-files-container li.selected a {
color: #5EB7E0;
}
.cblue {
color: #2890eb;
}
.sblue{
color: #91e069;
}
.kblue{
color: #c562fc;
}
.purple{
color: #8c03ab;
font-weight: 650;
}
.bgc{
background-color: #496968;
}
.marker{
background-color:#786060;
color: #faf26e;
}
.smtittle{
font-size: 20px;
}
</style>