# <h1>Learning 8-bit parity checking problem with MLP</h1>
###### tags: `深度學習` `Python` `大三`
<h2>目錄</h2>
<ul>
<li>目錄</li>
<li>
問題
<ul>
<li>問題內容</li>
<li>問題限制</li>
</ul>
</li>
<li>
解決過程
<ul>
<li>產生訓練資料以及測試資料</li>
<li>產生神經網絡模型</li>
<li>開始訓練模型</li>
<li>繪製學習曲線圖</li>
<li>顯示訓練後的結果</li>
</ul>
</li>
<li>比較兩層、三層、四層</li>
</ul>
<h2>問題</h2>
**問題內容：**<br>
設計一個MLP來學習解決判定8位元的奇偶數量問題，其問題是8位元的二進制輸入，檢測其中1的數量，若1的數量為奇數，則會輸出1，若反之1的數量為偶數，則會輸出0。
**問題限制：**<br>
1. 不能使用如Keras、Tensorflow、Pytorch、CNTK、MXNet等任何的深度學習框架<br>
2. 必須試驗兩層、三層、四層神經元網絡，並比較性能<br>
3. 需使用Sigmoid(x)、Relu(x)、Linear(x)以及Tanh(x)函數<br>
4. 需繪製Loss值的學習曲線圖，以及使用表格列出所有256個輸入的輸出以及停止時獲得的最終Loss值。
<h2>解決過程</h2>
**產生訓練資料以及測試資料**
在進行訓練以前，需要獲得訓練以及測試使用的資料，藉以下的程式碼可以產生256個二進制的8-bit訓練資料以及256個1-bit的訓練資料答案
```
data = [0,0,0,0,0,0,0,0]
print('([',end='')
count=0
while(True):
print("[",end='')
count1=0
for i in range(0,8,1):
if(data[i]==1):
count1+=1
#if(i!=7): #從此行開始以下四行可以產生8-bit訓練資料
#print(data[i],end=',')
#else:
#print(data[i],end='')
#if(count1%2==0): #此行開始下面四行可以產生1-bit的訓練資料答案
print('0',end='')
#else:
print('1',end='')
print("]",end='')
count+=1
if(count==256):
print('])')
break
print(',',end='')
data[7]+=1
j = 7
while(j>-1): #檢查是否進位(二進制)
if(data[j]==2):
data[j]=0
data[j-1]+=1
j-=1
else:
break
```
使用X來儲存產生的訓練資料，Y來儲存訓練資料用來比較Loss值的答案
```
x=np.array([[0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,1],[0,0,0,0,0,0,1,0],[0,0,0,0,0,0,1,1],[0,0,0,0,0,1,0,0],[0,0,0,0,0,1,0,1],[0,0,0,0,0,1,1,0],[0,0,0,0,0,1,1,1],[0,0,0,0,1,0,0,0],[0,0,0,0,1,0,0,1],[0,0,0,0,1,0,1,0],[0,0,0,0,1,0,1,1],[0,0,0,0,1,1,0,0],[0,0,0,0,1,1,0,1],[0,0,0,0,1,1,1,0],[0,0,0,0,1,1,1,1],[0,0,0,1,0,0,0,0],[0,0,0,1,0,0,0,1],[0,0,0,1,0,0,1,0],[0,0,0,1,0,0,1,1],[0,0,0,1,0,1,0,0],[0,0,0,1,0,1,0,1],[0,0,0,1,0,1,1,0],[0,0,0,1,0,1,1,1],[0,0,0,1,1,0,0,0],[0,0,0,1,1,0,0,1],[0,0,0,1,1,0,1,0],[0,0,0,1,1,0,1,1],[0,0,0,1,1,1,0,0],[0,0,0,1,1,1,0,1],[0,0,0,1,1,1,1,0],[0,0,0,1,1,1,1,1],[0,0,1,0,0,0,0,0],[0,0,1,0,0,0,0,1],[0,0,1,0,0,0,1,0],[0,0,1,0,0,0,1,1],[0,0,1,0,0,1,0,0],[0,0,1,0,0,1,0,1],[0,0,1,0,0,1,1,0],[0,0,1,0,0,1,1,1],[0,0,1,0,1,0,0,0],[0,0,1,0,1,0,0,1],[0,0,1,0,1,0,1,0],[0,0,1,0,1,0,1,1],[0,0,1,0,1,1,0,0],[0,0,1,0,1,1,0,1],[0,0,1,0,1,1,1,0],[0,0,1,0,1,1,1,1],[0,0,1,1,0,0,0,0],[0,0,1,1,0,0,0,1],[0,0,1,1,0,0,1,0],[0,0,1,1,0,0,1,1],[0,0,1,1,0,1,0,0],[0,0,1,1,0,1,0,1],[0,0,1,1,0,1,1,0],[0,0,1,1,0,1,1,1],[0,0,1,1,1,0,0,0],[0,0,1,1,1,0,0,1],[0,0,1,1,1,0,1,0],[0,0,1,1,1,0,1,1],[0,0,1,1,1,1,0,0],[0,0,1,1,1,1,0,1],[0,0,1,1,1,1,1,0],[0,0,1,1,1,1,1,1],[0,1,0,0,0,0,0,0],[0,1,0,0,0,0,0,1],[0,1,0,0,0,0,1,0],[0,1,0,0,0,0,1,1],[0,1,0,0,0,1,0,0],[0,1,0,0,0,1,0,1],[0,1,0,0,0,1,1,0],[0,1,0,0,0,1,1,1],[0,1,0,0,1,0,0,0],[0,1,0,0,1,0,0,1],[0,1,0,0,1,0,1,0],[0,1,0,0,1,0,1,1],[0,1,0,0,1,1,0,0],[0,1,0,0,1,1,0,1],[0,1,0,0,1,1,1,0],[0,1,0,0,1,1,1,1],[0,1,0,1,0,0,0,0],[0,1,0,1,0,0,0,1],[0,1,0,1,0,0,1,0],[0,1,0,1,0,0,1,1],[0,1,0,1,0,1,0,0],[0,1,0,1,0,1,0,1],[0,1,0,1,0,1,1,0],[0,1,0,1,0,1,1,1],[0,1,0,1,1,0,0,0],[0,1,0,1,1,0,0,1],[0,1,0,1,1,0,1,0],[0,1,0,1,1,0,1,1],[0,1,0,1,1,1,0,0],[0,1,0,1,1,1,0,1],[0,1,0,1,1,1,1,0],[0,1,0,1,1,1,1,1],[0,1,1,0,0,0,0,0],[0,1,1,0,0,0,0,1],[0,1,1,0,0,0,1,0],[0,1,1,0,0,0,1,1],[0,1,1,0,0,1,0,0],[0,1,1,0,0,1,0,1],[0,1,1,0,0,1,1,0],[0,1,1,0,0,1,1,1],[0,1,1,0,1,0,0,0],[0,1,1,0,1,0,0,1],[0,1,1,0,1,0,1,0],[0,1,1,0,1,0,1,1],[0,1,1,0,1,1,0,0],[0,1,1,0,1,1,0,1],[0,1,1,0,1,1,1,0],[0,1,1,0,1,1,1,1],[0,1,1,1,0,0,0,0],[0,1,1,1,0,0,0,1],[0,1,1,1,0,0,1,0],[0,1,1,1,0,0,1,1],[0,1,1,1,0,1,0,0],[0,1,1,1,0,1,0,1],[0,1,1,1,0,1,1,0],[0,1,1,1,0,1,1,1],[0,1,1,1,1,0,0,0],[0,1,1,1,1,0,0,1],[0,1,1,1,1,0,1,0],[0,1,1,1,1,0,1,1],[0,1,1,1,1,1,0,0],[0,1,1,1,1,1,0,1],[0,1,1,1,1,1,1,0],[0,1,1,1,1,1,1,1],[1,0,0,0,0,0,0,0],[1,0,0,0,0,0,0,1],[1,0,0,0,0,0,1,0],[1,0,0,0,0,0,1,1],[1,0,0,0,0,1,0,0],[1,0,0,0,0,1,0,1],[1,0,0,0,0,1,1,0],[1,0,0,0,0,1,1,1],[1,0,0,0,1,0,0,0],[1,0,0,0,1,0,0,1],[1,0,0,0,1,0,1,0],[1,0,0,0,1,0,1,1],[1,0,0,0,1,1,0,0],[1,0,0,0,1,1,0,1],[1,0,0,0,1,1,1,0],[1,0,0,0,1,1,1,1],[1,0,0,1,0,0,0,0],[1,0,0,1,0,0,0,1],[1,0,0,1,0,0,1,0],[1,0,0,1,0,0,1,1],[1,0,0,1,0,1,0,0],[1,0,0,1,0,1,0,1],[1,0,0,1,0,1,1,0],[1,0,0,1,0,1,1,1],[1,0,0,1,1,0,0,0],[1,0,0,1,1,0,0,1],[1,0,0,1,1,0,1,0],[1,0,0,1,1,0,1,1],[1,0,0,1,1,1,0,0],[1,0,0,1,1,1,0,1],[1,0,0,1,1,1,1,0],[1,0,0,1,1,1,1,1],[1,0,1,0,0,0,0,0],[1,0,1,0,0,0,0,1],[1,0,1,0,0,0,1,0],[1,0,1,0,0,0,1,1],[1,0,1,0,0,1,0,0],[1,0,1,0,0,1,0,1],[1,0,1,0,0,1,1,0],[1,0,1,0,0,1,1,1],[1,0,1,0,1,0,0,0],[1,0,1,0,1,0,0,1],[1,0,1,0,1,0,1,0],[1,0,1,0,1,0,1,1],[1,0,1,0,1,1,0,0],[1,0,1,0,1,1,0,1],[1,0,1,0,1,1,1,0],[1,0,1,0,1,1,1,1],[1,0,1,1,0,0,0,0],[1,0,1,1,0,0,0,1],[1,0,1,1,0,0,1,0],[1,0,1,1,0,0,1,1],[1,0,1,1,0,1,0,0],[1,0,1,1,0,1,0,1],[1,0,1,1,0,1,1,0],[1,0,1,1,0,1,1,1],[1,0,1,1,1,0,0,0],[1,0,1,1,1,0,0,1],[1,0,1,1,1,0,1,0],[1,0,1,1,1,0,1,1],[1,0,1,1,1,1,0,0],[1,0,1,1,1,1,0,1],[1,0,1,1,1,1,1,0],[1,0,1,1,1,1,1,1],[1,1,0,0,0,0,0,0],[1,1,0,0,0,0,0,1],[1,1,0,0,0,0,1,0],[1,1,0,0,0,0,1,1],[1,1,0,0,0,1,0,0],[1,1,0,0,0,1,0,1],[1,1,0,0,0,1,1,0],[1,1,0,0,0,1,1,1],[1,1,0,0,1,0,0,0],[1,1,0,0,1,0,0,1],[1,1,0,0,1,0,1,0],[1,1,0,0,1,0,1,1],[1,1,0,0,1,1,0,0],[1,1,0,0,1,1,0,1],[1,1,0,0,1,1,1,0],[1,1,0,0,1,1,1,1],[1,1,0,1,0,0,0,0],[1,1,0,1,0,0,0,1],[1,1,0,1,0,0,1,0],[1,1,0,1,0,0,1,1],[1,1,0,1,0,1,0,0],[1,1,0,1,0,1,0,1],[1,1,0,1,0,1,1,0],[1,1,0,1,0,1,1,1],[1,1,0,1,1,0,0,0],[1,1,0,1,1,0,0,1],[1,1,0,1,1,0,1,0],[1,1,0,1,1,0,1,1],[1,1,0,1,1,1,0,0],[1,1,0,1,1,1,0,1],[1,1,0,1,1,1,1,0],[1,1,0,1,1,1,1,1],[1,1,1,0,0,0,0,0],[1,1,1,0,0,0,0,1],[1,1,1,0,0,0,1,0],[1,1,1,0,0,0,1,1],[1,1,1,0,0,1,0,0],[1,1,1,0,0,1,0,1],[1,1,1,0,0,1,1,0],[1,1,1,0,0,1,1,1],[1,1,1,0,1,0,0,0],[1,1,1,0,1,0,0,1],[1,1,1,0,1,0,1,0],[1,1,1,0,1,0,1,1],[1,1,1,0,1,1,0,0],[1,1,1,0,1,1,0,1],[1,1,1,0,1,1,1,0],[1,1,1,0,1,1,1,1],[1,1,1,1,0,0,0,0],[1,1,1,1,0,0,0,1],[1,1,1,1,0,0,1,0],[1,1,1,1,0,0,1,1],[1,1,1,1,0,1,0,0],[1,1,1,1,0,1,0,1],[1,1,1,1,0,1,1,0],[1,1,1,1,0,1,1,1],[1,1,1,1,1,0,0,0],[1,1,1,1,1,0,0,1],[1,1,1,1,1,0,1,0],[1,1,1,1,1,0,1,1],[1,1,1,1,1,1,0,0],[1,1,1,1,1,1,0,1],[1,1,1,1,1,1,1,0],[1,1,1,1,1,1,1,1]])
y=np.array([[0],[1],[1],[0],[1],[0],[0],[1],[1],[0],[0],[1],[0],[1],[1],[0],[1],[0],[0],[1],[0],[1],[1],[0],[0],[1],[1],[0],[1],[0],[0],[1],[1],[0],[0],[1],[0],[1],[1],[0],[0],[1],[1],[0],[1],[0],[0],[1],[0],[1],[1],[0],[1],[0],[0],[1],[1],[0],[0],[1],[0],[1],[1],[0],[1],[0],[0],[1],[0],[1],[1],[0],[0],[1],[1],[0],[1],[0],[0],[1],[0],[1],[1],[0],[1],[0],[0],[1],[1],[0],[0],[1],[0],[1],[1],[0],[0],[1],[1],[0],[1],[0],[0],[1],[1],[0],[0],[1],[0],[1],[1],[0],[1],[0],[0],[1],[0],[1],[1],[0],[0],[1],[1],[0],[1],[0],[0],[1],[1],[0],[0],[1],[0],[1],[1],[0],[0],[1],[1],[0],[1],[0],[0],[1],[0],[1],[1],[0],[1],[0],[0],[1],[1],[0],[0],[1],[0],[1],[1],[0],[0],[1],[1],[0],[1],[0],[0],[1],[1],[0],[0],[1],[0],[1],[1],[0],[1],[0],[0],[1],[0],[1],[1],[0],[0],[1],[1],[0],[1],[0],[0],[1],[0],[1],[1],[0],[1],[0],[0],[1],[1],[0],[0],[1],[0],[1],[1],[0],[1],[0],[0],[1],[0],[1],[1],[0],[0],[1],[1],[0],[1],[0],[0],[1],[1],[0],[0],[1],[0],[1],[1],[0],[0],[1],[1],[0],[1],[0],[0],[1],[0],[1],[1],[0],[1],[0],[0],[1],[1],[0],[0],[1],[0],[1],[1],[0]])
```
<br>
**產生神經網絡模型**
會使用到四種函數(Sigmoid,Relu,Linear,Tanh)
```
class Sigmoid():
def __init__(self):
pass
def forward(self,x): #往前推所使用的Sigmoid函數
out = 1/(1+np.exp(-x))
self.o = out
return out
def backward(self,dout): #往回推所使用的Sigmoid導函數
dx = dout*self.o*(1-self.o)
return dx
class Tanh():
def __init__(self):
pass
def forward(self,x): #往前推所使用的Tanh函數
out = (1-np.exp(-x))/(1+np.exp(-x))
self.o = out
return out
def backward(self,out): #往回推所使用的Tanh導函數
dx = out * (1 - (self.o * self.o))
return dx
class Relu():
def __init__(self):
pass
def forward(self, x): #往前推所使用的Relu函數
self.mask = (x<=0)
out = x
out[out<=0] = 0
return out
def backward(self, dout): #往回推所使用的Relu導函數
dx = dout
dx[self.mask] = 0
return dx
class Linear(): #Linear函數
def __init__(self, m, n):
self.W, self.b = np.random.randn(m,n),np.random.randn(1,n) #隨機給予MxN個W和1xN個b
self.dW , self.db = None,None #dW,db設為none
def forward(self, x):
self.x = x
out = np.dot(x, self.W)+self.b #計算Wx+b放進out
return out
def backward(self, dout): #往回推計算學習後的dW和db
dx = np.dot(dout, self.W.T)
self.dW = np.dot(self.x.T, dout)
self.db = np.sum(dout, axis = 0)
return dx
```
還會使用Loss函數來計算與測試資料的差別
```
class Loss():
def __init__(self):
pass
def forward(self, y ,ybar): #y為算出的資料,ybar為訓練資料
self.ybar = ybar
return np.sum((y-ybar)**2)
def backward(self, dout): #往回推計算dy
dy = -(2*(y-self.ybar))
return dy
```
每多一層神經元，會多使用一層Linear函數和四選一的函數，為本題方便計算最後一層都為Sigmoid函數。
二層神經元的class和呼叫函數
```
class TwoLayer():
def __init__(self, m, n, o):
self.linear1 = Linear(m,n) #第一層
self.act1 = Tanh()
self.linear2 = Linear(n,o) #第二層
self.act2 = Sigmoid()
self.loss = Loss() #最後計算的Loss函數
self.last_dW1, self.last_db1 = 0 , 0 #初始化第一層最終的dW,db
self.last_dW2, self.last_db2 = 0 , 0 #初始化第二層最終的dW,db
def forward(self, x): #每層從Linear開始跑而後經過每層的函數
x = self.linear1.forward(x)
x = self.act1.forward(x)
x = self.linear2.forward(x)
self.ybar = self.act2.forward(x)
return self.ybar
def backward(self, y): #從最後開始由導函數按照順序回頭調整資料
self.L = self.loss.forward(y,self.ybar)
g = self.loss.backward(1)
g = self.act2.backward(g)
g = self.linear2.backward(g)
g = self.act1.backward(g)
g = self.linear1.backward(g)
def update(self, eta, alpha): #使用eta和alpha來調整每層的Linear和最終W,b
self.linear1.W = self.linear1.W-eta*self.linear1.dW + alpha*self.last_dW1
self.linear1.b = self.linear1.b-eta*self.linear1.db + alpha*self.last_db1
self.last_dW1 = eta*self.linear1.dW
self.last_db1 = eta*self.linear1.db
self.linear2.W = self.linear2.W-eta*self.linear2.dW + alpha*self.last_dW2
self.linear2.b = self.linear2.b-eta*self.linear2.db + alpha*self.last_db2
self.last_dW2 = eta*self.linear2.dW
self.last_db2 = eta*self.linear2.db
model = TwoLayer(8, 10, 1) #建立模型，初始進入8-bit，經過中間的神經元，最後輸出1-bit
max_epochs, chk_epochs = 20000, 500 #最多20000次，每500次輸出一次
last_dW, last_db = 0, 0
eta, alpha = 0.02, 0.07 #eta是學習效率係數,alpha為衝量係數
Epochs_data=[] #儲存Epochs的List
Loss_data=[] #儲存Loss值的List
for e in range(max_epochs):
model.forward(X)
model.backward(y)
model.update(eta, alpha)
if (e+1)%chk_epochs == 0:
print('epochs = %3d,loss = %.6f'%(e+1, model.L))
Loss_data.append(model.L)
Epochs_data.append(e+1)
plt.plot(Epochs_data,Loss_data,label='Loss_data',color='blue') #繪製曲線圖
plt.legend()
plt.show() #輸出曲線圖
```
三層神經元的class和呼叫函數
```
class ThreeLayer():
def __init__(self, m, n, o, p):
self.linear1 = Linear(m,n) #第一層
self.act1 = Tanh()
self.linear2 = Linear(n,o) #第二層
self.act2 = Relu()
self.linear3 = Linear(o,p) #第三層
self.act3 = Sigmoid()
self.loss = Loss() #最後計算的Loss函數
self.last_dW1, self.last_db1 = 0 , 0 #初始化第一層最終的dW,db
self.last_dW2, self.last_db2 = 0 , 0 #初始化第二層最終的dW,db
self.last_dW3, self.last_db3 = 0 , 0 #初始化第三層最終的dW,db
def forward(self, x): #每層從Linear開始跑而後經過每層的函數
x = self.linear1.forward(x)
x = self.act1.forward(x)
x = self.linear2.forward(x)
x = self.act2.forward(x)
x = self.linear3.forward(x)
self.ybar = self.act3.forward(x)
return self.ybar
def backward(self, y): #從最後開始由導函數按照順序回頭調整資料
self.L = self.loss.forward(y,self.ybar)
g = self.loss.backward(1)
g = self.act3.backward(g)
g = self.linear3.backward(g)
g = self.act2.backward(g)
g = self.linear2.backward(g)
g = self.act1.backward(g)
g = self.linear1.backward(g)
def update(self, eta, alpha): #使用eta和alpha來調整每層的Linear和最終W,b
self.linear1.W = self.linear1.W-eta*self.linear1.dW + alpha*self.last_dW1
self.linear1.b = self.linear1.b-eta*self.linear1.db + alpha*self.last_db1
self.last_dW1 = eta*self.linear1.dW
self.last_db1 = eta*self.linear1.db
self.linear2.W = self.linear2.W-eta*self.linear2.dW + alpha*self.last_dW2
self.linear2.b = self.linear2.b-eta*self.linear2.db + alpha*self.last_db2
self.last_dW2 = eta*self.linear2.dW
self.last_db2 = eta*self.linear2.db
self.linear3.W = self.linear3.W-eta*self.linear3.dW + alpha*self.last_dW3
self.linear3.b = self.linear3.b-eta*self.linear3.db + alpha*self.last_db3
self.last_dW3 = eta*self.linear3.dW
self.last_db3 = eta*self.linear3.db
model = ThreeLayer(8, 10, 10, 1)#建立模型，初始進入8-bit，經過中間的神經元，最後輸出1-bit
max_epochs, chk_epochs = 20000, 1000 #最多20000次，每1000次輸出一次
last_dW, last_db = 0, 0
eta, alpha = 0.001, 0.5 #eta是學習效率係數,alpha為衝量係數
Epochs_data=[] #儲存Epochs值的List
Loss_data=[] #儲存Loss值的List
for e in range(max_epochs):
model.forward(x)
model.backward(y)
model.update(eta, alpha)
if (e+1)%chk_epochs == 0:
print('epochs = %3d,loss = %.6f'%(e+1, model.L))
Loss_data.append(model.L)
Epochs_data.append(e+1)
plt.plot(Epochs_data,Loss_data,label='Loss_data',color='blue') #繪製曲線圖
plt.legend()
plt.show
```
四層神經元的class和呼叫函數
```
class FourLayer():
def __init__(self, m, n, o, p, q):
self.linear1 = Linear(m,n) #第一層
self.act1 = Tanh()
self.linear2 = Linear(n,o) #第二層
self.act2 = Relu()
self.linear3 = Linear(o,p) #第三層
self.act3 = Relu()
self.linear4 = Linear(p,q) #第四層
self.act4 = Sigmoid()
self.loss = Loss() #最後計算的Loss函數
self.last_dW1, self.last_db1 = 0 , 0 #初始化第一層最終的dW,db
self.last_dW2, self.last_db2 = 0 , 0 #初始化第二層最終的dW,db
self.last_dW3, self.last_db3 = 0 , 0 #初始化第三層最終的dW,db
self.last_dW4, self.last_db4 = 0 , 0 #初始化第四層最終的dW,db
def forward(self, x): #每層從Linear開始跑而後經過每層的函數
x = self.linear1.forward(x)
x = self.act1.forward(x)
x = self.linear2.forward(x)
x = self.act2.forward(x)
x = self.linear3.forward(x)
x = self.act3.forward(x)
x = self.linear4.forward(x)
self.ybar = self.act4.forward(x)
return self.ybar
def backward(self, y): #從最後開始由導函數按照順序回頭調整資料
self.L = self.loss.forward(y,self.ybar)
g = self.loss.backward(1)
g = self.act4.backward(g)
g = self.linear4.backward(g)
g = self.act3.backward(g)
g = self.linear3.backward(g)
g = self.act2.backward(g)
g = self.linear2.backward(g)
g = self.act1.backward(g)
g = self.linear1.backward(g)
def update(self, eta, alpha): #使用eta和alpha來調整每層的Linear和最終W,b
self.linear1.W = self.linear1.W-eta*self.linear1.dW + alpha*self.last_dW1
self.linear1.b = self.linear1.b-eta*self.linear1.db + alpha*self.last_db1
self.last_dW1 = eta*self.linear1.dW
self.last_db1 = eta*self.linear1.db
self.linear2.W = self.linear2.W-eta*self.linear2.dW + alpha*self.last_dW2
self.linear2.b = self.linear2.b-eta*self.linear2.db + alpha*self.last_db2
self.last_dW2 = eta*self.linear2.dW
self.last_db2 = eta*self.linear2.db
self.linear3.W = self.linear3.W-eta*self.linear3.dW + alpha*self.last_dW3
self.linear3.b = self.linear3.b-eta*self.linear3.db + alpha*self.last_db3
self.last_dW3 = eta*self.linear3.dW
self.last_db3 = eta*self.linear3.db
self.linear4.W = self.linear4.W-eta*self.linear4.dW + alpha*self.last_dW4
self.linear4.b = self.linear4.b-eta*self.linear4.db + alpha*self.last_db4
self.last_dW4 = eta*self.linear4.dW
self.last_db4 = eta*self.linear4.db
model = FourLayer(8, 10, 10, 10, 1)#建立模型，初始進入8-bit，經過中間的神經元，最後輸出1-bit
max_epochs, chk_epochs = 10000, 500 #最多10000次，每500次輸出一次
last_dW, last_db = 0, 0
eta, alpha = 0.001 , 0.6 #eta是學習效率係數,alpha為衝量係數
Loss_data=[] #儲存Loss值的List
Epochs_data=[] #儲存Epochs值的List
for e in range(max_epochs):
model.forward(x)
model.backward(y)
model.update(eta, alpha)
if (e+1)%chk_epochs == 0:
print('epochs = %3d,loss = %.6f'%(e+1, model.L))
Loss_data.append(model.L)
Epochs_data.append(e+1)
plt.plot(Epochs_data,Loss_data,label='Loss_data',color='blue') #繪製曲線圖
plt.legend()
plt.show()
```
<br>
**開始訓練模型**
二層的訓練過程
```
epochs = 500,loss = 64.006983
epochs = 1000,loss = 63.960385
epochs = 1500,loss = 41.666004
epochs = 2000,loss = 11.239978
epochs = 2500,loss = 9.035393
epochs = 3000,loss = 8.327138
epochs = 3500,loss = 7.651564
epochs = 4000,loss = 2.703534
epochs = 4500,loss = 2.179452
epochs = 5000,loss = 2.075702
epochs = 5500,loss = 2.031646
epochs = 6000,loss = 2.006583
epochs = 6500,loss = 1.989978
epochs = 7000,loss = 1.977873
epochs = 7500,loss = 1.968416
epochs = 8000,loss = 1.960591
epochs = 8500,loss = 1.953756
epochs = 9000,loss = 1.947417
epochs = 9500,loss = 1.941092
epochs = 10000,loss = 1.934153
epochs = 10500,loss = 1.925638
epochs = 11000,loss = 1.914327
epochs = 11500,loss = 1.900473
epochs = 12000,loss = 1.886242
epochs = 12500,loss = 1.868232
epochs = 13000,loss = 1.792538
epochs = 13500,loss = 1.548158
epochs = 14000,loss = 1.300410
epochs = 14500,loss = 1.124723
epochs = 15000,loss = 1.016667
epochs = 15500,loss = 0.827030
epochs = 16000,loss = 0.760825
epochs = 16500,loss = 0.724202
epochs = 17000,loss = 0.699248
epochs = 17500,loss = 0.680387
epochs = 18000,loss = 0.665197
epochs = 18500,loss = 0.652476
epochs = 19000,loss = 0.641561
epochs = 19500,loss = 0.632050
epochs = 20000,loss = 0.623675
```
---
三層神經元的訓練過程
```
epochs = 1000,loss = 63.719888
epochs = 2000,loss = 63.422269
epochs = 3000,loss = 63.050350
epochs = 4000,loss = 61.718144
epochs = 5000,loss = 59.181346
epochs = 6000,loss = 55.746686
epochs = 7000,loss = 51.429207
epochs = 8000,loss = 45.477023
epochs = 9000,loss = 39.902673
epochs = 10000,loss = 34.967170
epochs = 11000,loss = 28.262937
epochs = 12000,loss = 22.592299
epochs = 13000,loss = 18.526720
epochs = 14000,loss = 14.513283
epochs = 15000,loss = 9.566735
epochs = 16000,loss = 6.163954
epochs = 17000,loss = 3.994223
epochs = 18000,loss = 2.747954
epochs = 19000,loss = 2.063435
epochs = 20000,loss = 1.621340
```
---
四層神經元的訓練過程
```
epochs = 500,loss = 64.850557
epochs = 1000,loss = 60.814486
epochs = 1500,loss = 54.855777
epochs = 2000,loss = 41.767946
epochs = 2500,loss = 25.836491
epochs = 3000,loss = 14.888509
epochs = 3500,loss = 4.237126
epochs = 4000,loss = 2.617696
epochs = 4500,loss = 2.017337
epochs = 5000,loss = 1.709142
epochs = 5500,loss = 1.513393
epochs = 6000,loss = 1.391095
epochs = 6500,loss = 1.311712
epochs = 7000,loss = 1.254371
epochs = 7500,loss = 1.212571
epochs = 8000,loss = 1.181797
epochs = 8500,loss = 1.158027
epochs = 9000,loss = 1.138851
epochs = 9500,loss = 1.123523
epochs = 10000,loss = 1.110921
```
<br>
**繪製學習曲線圖**
使用matplotlib模組可以繪製學習曲線圖，使用以下程式碼儲存資料、繪圖、印出曲線圖
```
import matplotlib.pyplot as plt
Loss_data=[] #儲存Epochs值的List
Epochs_data=[] #儲存Loss值的List
for e in range(max_epochs):
model.forward(x)
model.backward(y)
model.update(eta, alpha)
if (e+1)%chk_epochs == 0:
print('epochs = %3d,loss = %.6f'%(e+1, model.L))
Loss_data.append(model.L)
Epochs_data.append(e+1)
plt.plot(Epochs_data,Loss_data,label='Loss_data',color='blue') #繪製曲線圖
plt.legend()
plt.show()
```
二層的學習曲線圖
---
三層的學習曲線圖
---
四層的學習曲線圖
<br>
**顯示訓練後結果**
二層最後的256個輸出結果
```
5.06565127e-03 9.94308914e-01 9.97965508e-01 3.05645022e-03
9.97951422e-01 3.06003186e-03 1.54950197e-02 9.98107911e-01
9.94381568e-01 3.00221062e-02 3.28890245e-03 9.93888754e-01
3.29298956e-03 9.93877828e-01 9.97919634e-01 2.23539564e-03
9.99910383e-01 2.71903334e-02 6.26887770e-03 9.99863328e-01
6.26610399e-03 9.99863659e-01 9.95304535e-01 5.55687840e-03
2.77555471e-02 4.88225575e-01 9.99878940e-01 2.81686918e-02
9.99879231e-01 2.81938445e-02 5.53988538e-03 9.99746427e-01
9.98484179e-01 1.52990421e-02 1.58075755e-02 9.97735420e-01
1.58128244e-02 9.97742132e-01 9.62548091e-01 7.39970498e-03
1.41027681e-02 9.95621978e-01 9.97909736e-01 1.46640927e-02
9.97915683e-01 1.45629891e-02 6.87586063e-03 9.96495564e-01
7.57311122e-03 9.99533641e-01 9.90469364e-01 5.71818308e-03
9.90459812e-01 5.72363275e-03 3.33825045e-03 9.90266732e-01
9.99657582e-01 1.52537580e-02 5.86530564e-03 9.98266819e-01
5.87126645e-03 9.98285305e-01 9.90113502e-01 4.61414015e-03
9.97994277e-01 3.02114883e-03 1.55149429e-02 9.98147905e-01
1.55097680e-02 9.98135018e-01 9.62231851e-01 7.64591856e-03
3.24967827e-03 9.93849946e-01 9.97962706e-01 2.21338750e-03
9.97948800e-01 2.21547854e-03 7.04860619e-03 9.98004674e-01
6.25459002e-03 9.99861577e-01 9.95318003e-01 5.55148823e-03
9.95314039e-01 5.55173119e-03 4.19589370e-03 9.95084986e-01
9.99877426e-01 2.80794041e-02 5.53390395e-03 9.99740821e-01
5.53407409e-03 9.99741541e-01 9.95024138e-01 5.09987411e-03
1.57982273e-02 9.97717595e-01 9.62546954e-01 7.39451415e-03
9.62546937e-01 7.39630775e-03 8.11158251e-02 9.82342767e-01
9.97893780e-01 1.48132451e-02 6.87045713e-03 9.96449975e-01
6.87226859e-03 9.96463353e-01 9.83124936e-01 3.80364655e-03
9.90488601e-01 5.70005201e-03 3.33985235e-03 9.90297152e-01
3.33942635e-03 9.90287193e-01 9.85775927e-01 2.38689147e-03
5.84502694e-03 9.98193767e-01 9.90144455e-01 4.59766912e-03
9.90134311e-01 4.60311760e-03 2.30093542e-03 9.89836720e-01
9.99011151e-01 1.11081633e-02 2.10845968e-03 9.98383910e-01
2.12621651e-03 9.98379169e-01 9.91084914e-01 9.47562582e-04
1.09317856e-02 9.40291138e-01 9.98412945e-01 1.18786304e-02
9.98408304e-01 1.18667542e-02 1.10908125e-03 9.97228961e-01
8.78837265e-03 9.39570400e-01 9.99745587e-01 7.67195866e-03
9.99744301e-01 7.70083729e-03 1.39183665e-03 9.99550352e-01
9.39470959e-01 5.60921041e-01 8.23351166e-03 9.60476406e-01
8.26426797e-03 9.60489681e-01 9.99563822e-01 7.24944411e-03
7.04162795e-03 9.99258320e-01 9.91113407e-01 5.77328971e-03
9.91094791e-01 5.78930033e-03 3.49776529e-03 9.91354895e-01
9.99266755e-01 6.02184378e-03 5.99615977e-03 9.98790007e-01
6.01302034e-03 9.98784491e-01 9.91096992e-01 5.41251859e-03
9.99622621e-01 1.42631682e-02 2.63952188e-03 9.98824236e-01
2.63602473e-03 9.98819382e-01 9.86148598e-01 1.63072999e-03
1.38244041e-02 9.58712010e-01 9.98937813e-01 1.57629586e-02
9.98933216e-01 1.57328770e-02 1.66647590e-03 9.95438624e-01
2.05867091e-03 9.98373107e-01 9.91146930e-01 9.23035163e-04
9.91142942e-01 9.29705652e-04 1.39150442e-02 9.92769434e-01
9.98402827e-01 1.19039376e-02 1.07930358e-03 9.97212281e-01
1.08739349e-03 9.97202179e-01 9.92483419e-01 5.88406506e-04
9.99744816e-01 7.59257700e-03 1.40600519e-03 9.99548313e-01
1.39817216e-03 9.99546303e-01 9.93974095e-01 1.04281485e-03
8.14916714e-03 9.60542190e-01 9.99562405e-01 7.14798780e-03
9.99560324e-01 7.17428570e-03 9.96317180e-04 9.99032923e-01
9.91174406e-01 5.74107222e-03 3.50799738e-03 9.91409343e-01
3.50510770e-03 9.91399727e-01 9.81239066e-01 2.71148160e-03
5.96167235e-03 9.98773843e-01 9.91161743e-01 5.37782286e-03
9.91149085e-01 5.39145587e-03 2.58686648e-03 9.90808356e-01
2.63152653e-03 9.98806099e-01 9.86134573e-01 1.63037810e-03
9.86159854e-01 1.62823925e-03 3.21099694e-02 9.87536218e-01
9.98922385e-01 1.58243357e-02 1.66527561e-03 9.95346922e-01
1.66328399e-03 9.95329741e-01 9.87475850e-01 1.07434743e-03
```
最後的Loss值
```
epochs = 20000,loss = 0.623675
```
三層最後的256個輸出結果
```
6.30075104e-04 9.87412179e-01 9.97117168e-01 3.56296033e-05
9.63390563e-01 4.68022487e-02 1.34084508e-02 9.89557079e-01
8.35154986e-01 6.69495123e-03 9.99585316e-02 9.97631620e-01
2.13484637e-01 9.57111657e-01 7.75058083e-01 2.45574534e-02
9.74597353e-01 1.38918482e-01 6.22203548e-06 9.74717073e-01
2.71162348e-02 9.65643253e-01 9.96212155e-01 4.03896462e-02
2.53471433e-03 9.90987883e-01 9.97573925e-01 4.69151617e-05
9.88048753e-01 4.61275486e-02 7.45971493e-03 9.92068555e-01
9.74268745e-01 1.94253525e-02 2.95491464e-02 9.96434522e-01
1.22853239e-01 8.40082802e-01 9.82182573e-01 1.71816247e-02
1.22000009e-01 9.82712414e-01 8.64265759e-01 3.37232765e-02
9.99234095e-01 6.45021671e-02 1.45806577e-01 9.88962635e-01
1.06144100e-02 9.20164391e-01 9.84245857e-01 8.39425179e-03
8.53883597e-01 1.28449766e-01 8.56537796e-02 9.38981574e-01
9.96691496e-01 3.18502087e-02 8.90232817e-02 9.97967577e-01
1.07574241e-01 7.95194617e-01 9.92530909e-01 2.51606709e-02
9.59708810e-01 4.99763937e-04 3.39686518e-02 9.96489389e-01
1.71227370e-02 9.86995226e-01 9.94410783e-01 2.63571317e-04
2.18778152e-01 9.93244780e-01 9.99673723e-01 3.84390956e-02
7.39620680e-01 2.39471871e-02 2.34738818e-01 9.06066573e-01
1.10060601e-04 8.94033102e-01 9.23254653e-01 9.22028337e-02
9.73894395e-01 8.99798476e-02 1.26990222e-04 9.91079817e-01
9.67602686e-01 7.18839635e-04 4.28317178e-02 9.96499401e-01
1.53128466e-02 9.91985855e-01 9.83920947e-01 2.68787116e-04
4.37653790e-02 9.92884952e-01 8.77868197e-01 6.27593971e-04
9.65425401e-01 1.54079985e-03 1.56875018e-02 9.97617759e-01
7.97435935e-01 2.89314939e-02 2.03215120e-01 9.95167301e-01
1.75053343e-01 9.93607770e-01 7.63943454e-01 3.76344869e-02
9.99009800e-01 4.43524598e-02 6.08078923e-04 9.39688778e-01
3.05140346e-03 9.20493979e-01 9.70174022e-01 4.61689552e-02
7.51395310e-02 9.96904065e-01 9.07502593e-01 1.13025377e-03
9.60214027e-01 2.34497439e-03 4.46759063e-02 9.97337897e-01
9.88963040e-01 6.87358172e-02 3.74936278e-05 8.95142609e-01
4.51068312e-02 9.46631402e-01 9.93474475e-01 1.39734387e-01
7.90602409e-03 9.93878149e-01 9.96953220e-01 6.90780210e-04
9.53636603e-01 6.17013730e-02 3.17166385e-02 9.96108884e-01
1.34088278e-01 8.83844367e-01 9.76455839e-01 1.09119317e-01
9.66511240e-01 1.42986182e-01 3.42605307e-02 8.15934712e-01
9.92261735e-01 5.86817862e-02 3.88068750e-05 8.66032301e-01
5.60372440e-02 9.65814015e-01 9.93808772e-01 1.24843786e-01
2.33492448e-02 9.40702178e-01 9.98161927e-01 4.95981633e-02
8.15652539e-01 1.59850806e-01 2.06672988e-02 9.32716600e-01
9.82424289e-01 7.27609196e-02 4.70132966e-02 9.97055771e-01
9.58601241e-02 8.16530316e-01 9.95646545e-01 1.22782504e-02
9.31554308e-01 1.53013999e-01 1.01127268e-02 9.10443331e-01
1.25822924e-01 9.72867443e-01 9.59248228e-01 5.01153526e-02
4.90499189e-02 9.51884677e-01 9.98776160e-01 3.78427039e-02
8.23120290e-01 1.62808760e-01 3.81261163e-02 9.17998435e-01
6.08108342e-04 9.06682062e-01 9.96970308e-01 7.92051458e-02
9.93375228e-01 8.66272806e-02 3.04936027e-04 9.09520385e-01
9.93839535e-01 1.32330704e-03 4.47108897e-02 9.49813719e-01
2.81502079e-02 9.93879296e-01 9.19055578e-01 6.01018610e-03
8.99412641e-01 2.19016157e-01 9.08974020e-02 8.27809174e-01
8.72330236e-02 8.08258048e-01 9.93486455e-01 8.99356052e-02
7.65079928e-04 8.89231610e-01 9.96716823e-01 1.02965756e-01
9.95548239e-01 7.88945721e-02 3.40673327e-04 9.03977244e-01
9.91093959e-01 1.17508576e-01 9.95902527e-04 9.98247318e-01
2.30885543e-03 9.57274482e-01 9.99004351e-01 5.10899280e-02
5.46362551e-02 9.98462364e-01 9.93049633e-01 1.62383681e-03
9.95703085e-01 2.80585660e-03 2.75710096e-02 9.97874750e-01
4.88903074e-02 7.77309402e-01 9.53106800e-01 9.54789746e-02
9.24408857e-01 1.00404942e-01 5.23241326e-02 9.54147637e-01
9.98151306e-01 8.90220385e-02 1.13966581e-03 9.97893411e-01
3.73579608e-03 9.47928508e-01 9.98803848e-01 3.08694304e-02
```
最後的Loss值
```
epochs = 20000,loss = 1.621340
```
四層最後的256個輸出結果
```
0.02703342 0.9997266 0.96434816 0.00872148 0.99977255 0.00588859
0.01631082 0.99975773 0.98792582 0.00676656 0.00460467 0.99356196
0.00804919 0.97464623 0.99126323 0.01150138 0.99964793 0.00698693
0.01247588 0.99967225 0.00721424 0.96988608 0.99974821 0.00498008
0.00870172 0.99824273 0.97568426 0.00630293 0.99905684 0.00924758
0.00798213 0.99884483 0.99155919 0.00388602 0.00293092 0.99610998
0.0059407 0.99598387 0.99776406 0.0030091 0.00364455 0.98963164
0.97601486 0.00291671 0.99030807 0.01258211 0.00321267 0.98254445
0.0050546 0.9977336 0.99927981 0.0059216 0.99713916 0.03914043
0.00411022 0.99840139 0.99866663 0.02138521 0.00344153 0.99851854
0.02090338 0.99847959 0.99824453 0.01459646 0.99973362 0.00545886
0.00859084 0.99984869 0.00625143 0.98778296 0.99984151 0.00402625
0.00715914 0.97725328 0.99331943 0.01658061 0.97093919 0.01520107
0.01465205 0.97306428 0.00750264 0.97927037 0.99976247 0.00485207
0.98090426 0.0662089 0.004939 0.98206748 0.99866744 0.01031971
0.00716142 0.99130063 0.01044761 0.92891994 0.99865188 0.00714822
0.00467602 0.99653061 0.99658791 0.00263037 0.99650455 0.08853935
0.00269588 0.99746052 0.98893965 0.01309402 0.00315638 0.98427122
0.01817288 0.94914373 0.98477915 0.00755183 0.99804299 0.04616213
0.00654843 0.99892273 0.03732178 0.87580975 0.99868447 0.0487141
0.02114952 0.98169323 0.9988512 0.01296707 0.9975096 0.06924399
0.01331219 0.99528152 0.99519531 0.01352103 0.99987886 0.99881119
0.02337009 0.99423919 0.99899635 0.00734103 0.01454398 0.98007931
0.94553297 0.01189111 0.97101532 0.01894722 0.02000454 0.98996579
0.02632993 0.99680696 0.99726402 0.02668914 0.9969222 0.01536692
0.0261257 0.99654932 0.98303638 0.0192096 0.04556013 0.97588533
0.01882383 0.98286538 0.98621335 0.0324123 0.01818293 0.99947004
0.97237642 0.01090769 0.99960236 0.00596079 0.02364898 0.99965769
0.99390101 0.00779484 0.00682176 0.99293197 0.00823552 0.97078533
0.99362855 0.01333203 0.99946338 0.00763562 0.01301885 0.99966069
0.01008159 0.96917283 0.99970943 0.00488672 0.00717558 0.98677254
0.97828321 0.00646851 0.98243167 0.0114093 0.00625458 0.98526655
0.00706674 0.99390027 0.99821516 0.0071698 0.99220796 0.00975581
0.00647336 0.995081 0.98571092 0.01893768 0.01143141 0.99113047
0.01081838 0.96215191 0.98979431 0.04723369 0.99552842 0.01039407
0.02197644 0.99682124 0.00814605 0.96053097 0.99673047 0.01395353
0.01501896 0.97971788 0.97967936 0.03430472 0.97230421 0.01443078
0.02835657 0.97699346 0.99968362 0.00497827 0.00884332 0.99970753
0.00780441 0.98663913 0.999745 0.00311976 0.01089792 0.97027835
0.98937393 0.02138175 0.95355966 0.01682119 0.01799441 0.97553557
0.00944472 0.97909898 0.99976587 0.00453673 0.97321851 0.06182825
0.0048744 0.9878081 0.98286013 0.01243165 0.00751149 0.98703873
0.01452024 0.9313321 0.98630306 0.00754203
```
最後的Loss值
```
epochs = 10000,loss = 1.110921
```
**比較兩層、三層、四層**
在兩層、三層、四層的程式碼中，eta以及alpha值的調整是需要比較的地方，當層數比較少時，學習效率係數和衝量係數可以比較大，一樣可以得到較小的Loss值，但是容易浮動，有較高的機率會得到很高的Loss值。
但相反的，在層數越高的神經元網路中，學習效率係數和衝量係數會越來越小，但是只要調整成功了，Loss值的浮動不會像少層的一樣浮動大，只會在小範圍中浮動。
Sign in with Wallet
Connect another wallet