---
# System prepended metadata

title: 'Artificial Neural Network,ANN'
tags: [111course]

---

---
title: Artificial Neural Network,ANN
tags: 111course
description: Use `{%hackmd theme-dark %}` syntax to include this theme.
---
#  ***<div id="animation_title">Artificial Neural Network</div>*** 
<font font-size:29px>Edited by $\mathfrak{Warren}$ </font>
:::spoiler 文章目錄
[toc]
:::
## What is <span class="sblue">Neural</span> <span class="cblue">Network</span>?
### *<span class="sblue">Neural</span>*: what are the neurons?
![](https://sds-platform-private.s3-us-east-2.amazonaws.com/uploads/47_blog_image_2.png =90%x)
+ Neuron<span class="kblue">(神經元)</span>$\rightarrow$thing that holds a number(activation).
+ Activation<span class="kblue">(激勵值)</span>$\rightarrow$every activaton in neuron in layers would effect the next layer.

### *<span class="cblue">Network</span>*: how are they connected?
![](https://ithelp.ithome.com.tw/upload/images/20171213/20001976b0pAMidGtl.jpg =80%x)
+ layer(層)$\rightarrow$there are many neurons in a layer and layers conneted each other.
+ <font color="#fc9ae5">input layer</font>$\longrightarrow$<font color="#9aa9fc">hidden layer1</font>$\longrightarrow$<font color="#9aa9fc">hidden layer2</font>$\longrightarrow$...$\longrightarrow$<font color="#9afcd3">output layer</font>
+ But, why layers?$\rightarrow$simliar to *"devide and conquer"*
+$$
### What are the synapse doing?
+ synapes $\rightarrow$the weight <span class="kblue">(權重值)</span>
+ we calculate the  next neuron's activation by 
$$
w_1a_1+w_2a_2+w_3a_3+w_4a_4+...+w_na_n
$$
+ But the number we cauculated may be any number
+ **SO**, we apply the **activation function**<span class="kblue">(激勵函數)</span>
+ **activation function**, take an example, ***Sigmoid Function***
$$
\sigma(t)= \frac{1}{1+e^{-t}}
$$
+ Then we put the number we got into the function and would get
$$
\sigma(w_1a_1+w_2a_2+w_3a_3+w_4a_4+...+w_na_n)
$$
+ And the next step, we adjust this number by adding a barrier for it.
+ The barrier is called ***Bias*** which can let we choose the activation which is higher than others. 
+ And that would be 
$$
\sigma(w_1a_1+w_2a_2+w_3a_3+w_4a_4+...+w_na_n-b)
$$
+ To see the overall, we can write every neuron as a equation.
$$
\sigma\{\left[\begin{array}{cc} 
    w_{11} & w_{12} & w_{13} & \cdots&w_{1n}\\
    w_{21} & w_{22} & w_{23} & \cdots&w_{2n}\\
    w_{31} & w_{32} & w_{33} & \cdots&w_{3n}\\
    \vdots & \vdots & \vdots & {}^\cdot\cdot_\cdot&\vdots\\
    w_{k1} & w_{k2} & w_{k3} & \cdots&w_{kn}
\end{array}\right]
\left[\begin{array}{cc} 
    a_{1} \\
    a_{2} \\
    a_{3} \\
    \vdots \\
    a_{k} 
\end{array}\right]+
\left[\begin{array}{cc} 
    b_{1} \\
    b_{2} \\
    b_{3} \\
    \vdots \\
    b_{k} 
\end{array}\right]\}
$$
+ Above equation can be written as :
$$
\sigma(W\overrightarrow{a}+\overrightarrow{b})
$$
## **Recent optimizers**
### Gradient descent, GD(梯度下降法)
<span class="sblue">*中文*</span>
[梯度下降法](http://surl.li/dxdoi)
[梯度最佳解相關算法](http://surl.li/dxdom)
[SGD, Momentum, AdaGrad, Adam Optimizer](https://reurl.cc/286exX)
<span class="sblue">*English*</span>
[An overview of gradient descent optimization algorithms](https://ruder.io/optimizing-gradient-descent/index.html#adam)
> 在神經網路中，不論是哪種網路，最後都是在找層和層之間的關係
> (參數，也就是層和層之間的權重)，而找參數的過程就稱為學習，
> 所以神經網路的目的就是不斷的更新參數，去最小化損失函數的值，
> 然後找到最佳解。
### Stochastic gradient descent, SGD(隨機梯度下降法)
### Momentum
### Adagrad
### RMSProp
### Adam























<style>
@-webkit-keyframes A
{
    0%  {  color:#fc4444;
    		padding-left:0px}
    5%	{	color:;
    		padding-left:5px;}
    10% {  color:#fc7844;
    		padding-left:10px}
    15%	{	color:;
    		padding-left:15px;}
    20% {  color:#fc7844;
    		padding-left:20px}
    25%	{	color:;
    		padding-left:25px;}
    30% {  color:#e7fc44;
    		padding-left:30px}
    35%	{	color:;
    		padding-left:35px;}
    40% {  color:#e7fc44;
    		padding-left:40px}
    45%	{	color:;
    		padding-left:45px;}
    50% {  color:#7bfc44;
    		padding-left:50px}
    55%	{	color:;
    		padding-left:45px;}
    60% {  color:#44fc9a;
    		padding-left:40px}
    65%	{	color:;
    		padding-left:35px;}
    70% {  color:#44fc9a;
    		padding-left:30px}
    75%	{	color:;
    		padding-left:25px;}
    80% {  color: #44fc9a;
    		padding-left:20px}
    85%	{	color:;
    		padding-left:15x;}
    90% {  color: #9144fc;
    		padding-left:10px}
    95%	{	color:;
    		padding-left:5px;}
   100% {  color: #fc44d7;
    		padding-left:0px}
}
#animation_title{
    animation: A 3s ease 0s infinite alternate;
    -webkit-animation: A 3s ease 0s infinite alternate;
    
  }


html, body, .ui-content {
    background-color: #333;
    color: #ccc;
}

.markdown-body h1,
.markdown-body h2,
.markdown-body h3,
.markdown-body h4,
.markdown-body h5,
.markdown-body h6 {
    color: #ddd;
}

.markdown-body h1,
.markdown-body h2 {
    border-bottom-color: #ffffff69;
}

.markdown-body h1 .octicon-link,
.markdown-body h2 .octicon-link,
.markdown-body h3 .octicon-link,
.markdown-body h4 .octicon-link,
.markdown-body h5 .octicon-link,
.markdown-body h6 .octicon-link {
    color: #fff;
}

.markdown-body img {
    background-color: transparent;
}

.ui-toc-dropdown .nav>.active:focus>a, .ui-toc-dropdown .nav>.active:hover>a, .ui-toc-dropdown .nav>.active>a {
    color: white;
    border-left: 2px solid white;
}

.expand-toggle:hover, 
.expand-toggle:focus, 
.back-to-top:hover, 
.back-to-top:focus, 
.go-to-bottom:hover, 
.go-to-bottom:focus {
    color: white;
}


.ui-toc-dropdown {
    background-color: #333;
}

.ui-toc-label.btn {
    background-color: #191919;
    color: white;
}

.ui-toc-dropdown .nav>li>a:focus, 
.ui-toc-dropdown .nav>li>a:hover {
    color: white;
    border-left: 1px solid white;
}

.markdown-body blockquote {
    color: #bcbcbc;
}

.markdown-body table tr {
    background-color: #5f5f5f;
}

.markdown-body table tr:nth-child(2n) {
    background-color: #4f4f4f;
}

.markdown-body code,
.markdown-body tt {
    color: #eee;
    background-color: rgba(230, 230, 230, 0.36);
}

a,
.open-files-container li.selected a {
    color: #5EB7E0;
}
    .cblue {
        color: #2890eb;
}
    .sblue{
        color: #91e069;
    }
    .kblue{
        color: #c562fc;
    }
    .purple{
        color: #8c03ab;
        font-weight: 650;
}   
    .bgc{
        background-color: #496968;
    }
    .marker{
        background-color:#786060;
        color: #faf26e;
    }
    .smtittle{
        font-size: 20px;
    }
</style>
