# CCU_2020ML HW01 多項式回歸練習
>author: 楊淨
>date: 2020/04/07
4105931 機器學習
Assignment #1 – Regression
Due Date : 04/07 23:59pm
本次作業主要是實作 Unit-5、p.14 頁的迴歸公式，就可以完成(a) -(d)題；
實作 Unit-7、p.11 頁的正規化公式，可以完成題(e)。
本次作業需要自己實作上述公式，請勿使用任何現成的 regression 函式。**
Use the linear model y = 2x +ε with zero-mean Gaussian noise ε∼ N(0,1)
to generate 20 data points with (equal spacing) x ∈ [−3, 3].
# part A 題目與作法:
(a) Perform linear regression. 20 data points are split into 15 training
samples and 5 testing samples (75% for training and 25% for testing).
Show the fitting plots of the training error, cross-validation errors
for both leave-one-out and five-fold, and testing errors.
1. 先將資料以`dataGen()`產出，並以x跟y接收後，再將他們以`partial()`分割成前75%訓練用資料及後25%測試用資料。
2. 接著再回歸方程式中利用x_train這筆訓練用資料`linear_regression(x_train,y_train)` 算出回歸線的各項權重，代回線性方程AX+B得出Y值
3. 利用`mse_error(y_train_pred, y_train)`算出誤差分數，便可以獲得訓練資料集中的mse_error 作為題目要的 training error。
4. 再利用寫好的`cross_Kfold_err(x, y,k)`可以自動將資料拆分成k等分進行Kfold驗證，分別訓練與驗證，算出其MSE_ERROR做為cross-validation errors，以此題來說，因為總資料量剛好是20，所以會拆分成20份19個訓練資料及1個測試資料即為LOOCV。
5. 同上利用`cross_Kfold_err(x, y,k)`，k代入5即為five-fold error
6. 最後利用拆分的test資料集 `y_test_pred = Fitted_func(x_test, W_lin[0], W_lin[1])` `mse_err = mse_error(y_test_pred, y_test)`計算出test error
7. 利用matplotlib庫繪製fitting plots
* **主要程式:**
```python=
def partA():
x, y = dataGen()
x_train,x_test = partial(x)
y_train,y_test = partial(y)
#將x,y帶入
W_lin =linear_regression(x_train,y_train)
print("linear_regression: y = " + str(W_lin[0]) + "x + " + str(W_lin[1]))
print("----------------train_err------------------")
y_train_pred = Fitted_func(x_train, W_lin[0], W_lin[1])
mse_err = mse_error(y_train_pred, y_train)
print(mse_err)
print("----------------cross_LOO_err------------------")
print(cross_Kfold_err(x, y,20))
print("----------------cross_five-fold_err------------------")
print(cross_Kfold_err(x, y,5))
print("----------------test_err------------------")
y_test_pred = Fitted_func(x_test, W_lin[0], W_lin[1])
mse_err = mse_error(y_test_pred, y_test)
print(mse_err)
plt.title("train data set")
plt.xlabel("x axis caption")
plt.ylabel("y axis caption")
plt.scatter(x_train, y_train, label="train-Data")
plt.scatter(x_test, y_test, label="test-Data")
plt.plot(x,Fitted_func(x, W_lin[0], W_lin[1]),label='Fitted function')
plt.legend()
plt.show()
return
```
* **生產資料**
```python=
def dataGen():
line = np.linspace(-3, 3, 20)
noise = np.random.normal(0, 1, line.shape)
testdata = 2 * line + noise
print(line)
print(testdata)
plt.title("origin data set")
plt.xlabel("x axis caption")
plt.ylabel("y axis caption")
plt.plot(line, testdata, '.')
plt.show()
return line,testdata
```
* **線性回歸**
```python=
def linear_regression(x, y):
x = np.concatenate(( x[:, np.newaxis],np.ones((x.shape[0], 1))), axis=1)
y = y[:, np.newaxis]
w_lin = np.matmul(np.matmul(np.linalg.inv(np.matmul(x.T, x)), x.T), y)
return w_lin
```
* **切分資料**
```python=
def partial(arr,start=15, rate=0.25):
k=math.floor(arr.size * rate)
test = arr[start: start + k]
pretrain=np.array([])
if (start != 0):
pretrain=arr[:start]
posttrain = arr[start + k :]
train=np.append(pretrain,posttrain)
return train, test
```
* **Mse_Error計算**
```python=
def mse_error(y_pred,y):
mse = np.mean((y_pred- y)** 2)
return mse
```
* **線性方程計算**
```python=
def Fitted_func(x, a, b):
return (a * x)+b
```
* **cross-validation K-fold計算**
```python=
def cross_Kfold_err(x, y, k):
Krate = 1 / k
startptr = 0
mse_err = 0
plt.title("cross_"+str(k)+"-fold_err")
plt.xlabel("x axis caption")
plt.ylabel("y axis caption")
for _ in range(k):
x_train, x_test = partial(x,startptr, rate=Krate)
y_train, y_test = partial(y, startptr, rate=Krate)
plt.scatter(x_train, y_train, label="train-Data")
plt.scatter(x_test, y_test, label="test-Data")
W_lin = linear_regression(x_train, y_train)
y_test_pred = Fitted_func(x_test, W_lin[0], W_lin[1])
K_err = mse_error(y_test_pred, y_test)
startptr = startptr + math.floor(x.size * Krate)
mse_err = mse_err + K_err
plt.plot(x,Fitted_func(x, W_lin[0], W_lin[1]),label=str(_)+'-th')
mse_err = mse_err / k
plt.legend()
plt.show()
return mse_err
```
# part A 結果:
| training error | leave-one-out error| five-fold error| testing error|
| -------- | -------- | -------- | -------- |
| 0.6056858377530092 | 1.4539140490809153 | 1.313419308133971 | 2.784714551445104 |
* **fitting plots of training and test error:**
* **fitting plots of leave-one-out error:**
* **fitting plots of five-fold error:**
# part A 討論
由於我分割資料時並非隨機散佈，而是全部擠一坨在尾端。
這比較接近預測未來走勢的回歸模型，在這種模型下，對於測試資料的error通常會特別大，因為沒有歷史資料的輔助，對於超出X範圍的點，他無法給予一個相關性較高的Y值，而交叉驗證的LOOCV、kTH_FOLD，會避免這類極端狀況的發生，所以壓低了ERROR值。
---
# part B 題目與作法:
(b) Perform polynomial regression with degree 5, 10 and 14, respectively.
For each case, show the fitting plots of the training error,
crossvalidation errors (both leave-one-out and five-fold) and
testing errors.
1. 做法與A基本上大同小異，主要是差在原本線性回歸改成了多項式回歸而已
2. 先將資料以`dataGen()`產出，並以x跟y接收後，再將他們以`partial()`分割成前75%訓練用資料及後25%測試用資料。
3. 接著再回歸方程式中利用x_train這筆訓練用資料`POLYNOMIAL_regression(x_train, y_train, 5)` 算出高次方回歸線的各項權重。
4. 利用`y_train_pred = Poly_func(x_train,W_poly )` `mse_error(y_train_pred, y_train)`算出誤差分數，便可以獲得訓練資料集中的mse_error 作為題目要的 training error。
5. 再利用寫好的`cross_Kfold_Poly_err(x, y,k,degree)`可以自動將資料拆分成k等分進行Kfold驗證，分別訓練與驗證，算出其MSE_ERROR做為cross-validation errors，以此題來說，因為總資料量剛好是20，所以會拆分成20份19個訓練資料及1個測試資料即為LOOCV。
6. 同上利用`cross_Kfold_Poly_err(x, y,k,degree)`，k代入5即為five-fold error
7. 最後用一開始拆分好的test資料集 `y_test_pred = Poly_func(x_test, W_lin[0], W_lin[1])` `mse_err = mse_error(y_test_pred, y_test)`計算出test error
8. 重複上述步驟將degree改成10、14
9. 利用matplotlib庫繪製fitting plots
* **主要程式:**
```python=
def partB():
x, y = dataGen()
x_train, x_test = partial(x)
print(x_train)
y_train, y_test = partial(y)
print(y_train)
#將x,y帶入
W_poly = POLYNOMIAL_regression(x_train, y_train, 5)
print("deg5_poly_regression: y = ")
for i in range(6):
print(str(W_poly[i]) + "x^" + str(5 - i))
print("----------------train_err------------------")
y_train_pred = Poly_func(x_train,W_poly )
mse_err = mse_error(y_train_pred, y_train)
print(mse_err)
print("----------------cross_LOO_err------------------")
print(cross_Kfold_Poly_err(x, y,20,5))
print("----------------cross_five-fold_err------------------")
print(cross_Kfold_Poly_err(x, y,5,5))
print("----------------test_err------------------")
y_test_pred = Poly_func(x_test,W_poly )
mse_err = mse_error(y_test_pred, y_test)
print(mse_err)
W_poly = POLYNOMIAL_regression(x_train, y_train, 10)
print("deg10_poly_regression: y = ")
for i in range(11):
print(str(W_poly[i]) + "x^"+str(10-i))
print("----------------train_err------------------")
y_train_pred = Poly_func(x_train,W_poly )
mse_err = mse_error(y_train_pred, y_train)
print(mse_err)
print("----------------cross_LOO_err------------------")
print(cross_Kfold_Poly_err(x, y,20,10))
print("----------------cross_five-fold_err------------------")
print(cross_Kfold_Poly_err(x, y,5,10))
print("----------------test_err------------------")
y_test_pred = Poly_func(x_test,W_poly )
mse_err = mse_error(y_test_pred, y_test)
print(mse_err)
W_poly = POLYNOMIAL_regression(x_train, y_train, 14)
print("deg14_poly_regression: y = ")
for i in range(15):
print(str(W_poly[i]) + "x^"+str(14-i))
print("----------------train_err------------------")
y_train_pred = Poly_func(x_train,W_poly )
mse_err = mse_error(y_train_pred, y_train)
print(mse_err)
print("----------------cross_LOO_err------------------")
print(cross_Kfold_Poly_err(x, y,20,14))
print("----------------cross_five-fold_err------------------")
print(cross_Kfold_Poly_err(x, y,5,14))
print("----------------test_err------------------")
y_test_pred = Poly_func(x_test,W_poly )
mse_err = mse_error(y_test_pred, y_test)
print(mse_err)
plt.title("poly_regression")
plt.xlabel("x axis caption")
plt.ylabel("y axis caption")
plt.ylim(-30, 30)
plt.scatter(x_train, y_train, label="train-Data")
plt.scatter(x_test, y_test, label="test-Data")
W_poly = POLYNOMIAL_regression(x_train, y_train, 5)
plt.plot(x, Poly_func(x, W_poly), label='deg5_poly_regression')
W_poly = POLYNOMIAL_regression(x_train, y_train, 10)
plt.plot(x,Poly_func(x,W_poly ),label='deg10_poly_regression')
W_poly = POLYNOMIAL_regression(x_train, y_train, 14)
plt.plot(x,Poly_func(x,W_poly ),label='deg14_poly_regression')
plt.legend()
plt.show()
return
```
* **生產資料**
```python=
def dataGen():
line = np.linspace(-3, 3, 20)
noise = np.random.normal(0, 1, line.shape)
testdata = 2 * line + noise
print(line)
print(testdata)
plt.title("origin data set")
plt.xlabel("x axis caption")
plt.ylabel("y axis caption")
plt.plot(line, testdata, '.')
plt.show()
return line,testdata
```
* **多項式回歸**
```python=
def POLYNOMIAL_regression(x, y, k):
ori_x=x[:, np.newaxis]
x = np.concatenate((x[:, np.newaxis], np.ones((x.shape[0], 1))), axis=1)
for i in range(2, k+1):
left_col = ori_x ** (i)
x = np.concatenate((left_col, x), axis=1)
y = y[:, np.newaxis]
#print(y)
w_POLY = np.matmul(np.matmul(np.linalg.inv(np.matmul(x.T, x)), x.T), y)
return w_POLY
```
* **切分資料**
```python=
def partial(arr,start=15, rate=0.25):
k=math.floor(arr.size * rate)
test = arr[start: start + k]
pretrain=np.array([])
if (start != 0):
pretrain=arr[:start]
posttrain = arr[start + k :]
train=np.append(pretrain,posttrain)
return train, test
```
* **Mse_Error計算**
```python=
def mse_error(y_pred,y):
mse = np.mean((y_pred- y)** 2)
return mse
```
* **多項式方程計算**
```python=
def Poly_func(x, w):
y=x*0
for i in range(w.size):
y = y + w[w.size-i-1] * (x ** i)
return y
```
* **cross-validation K-fold計算**
```python=
def cross_Kfold_Poly_err(x, y, k,deg):
Krate = 1 / k
startptr = 0
mse_err = 0
plt.title("cross_"+str(k)+"-fold_POLY_err")
plt.xlabel("x axis caption")
plt.ylabel("y axis caption")
plt.ylim(-10, 10)
for _ in range(k):
x_train, x_test = partial(x,startptr, rate=Krate)
y_train, y_test = partial(y,startptr, rate=Krate)
W_poly = POLYNOMIAL_regression(x_train, y_train, deg)
y_test_pred = Poly_func(x_test, W_poly)
K_err = mse_error(y_test_pred, y_test)
startptr = startptr + math.floor(x.size * Krate)
mse_err = mse_err + K_err
plt.scatter(x_train, y_train, label="train-Data")
plt.scatter(x_test, y_test, label="test-Data")
plt.plot(x, Poly_func(x, W_poly), label=str(_)+'TH deg-'+str(deg)+'_poly_regression')
plt.legend()
plt.show()
mse_err=mse_err/k
return mse_err
```
# part B 結果:
| degree | training error | leave-one-out error | five-fold error | testing error |
| ------ | -------------- | ------------------- | --------------- | ------------- |
| 5 |0.7159022520706798 | 1.6834928238066844 |187.44561217194774 |211.6211352155488|
| 10 |0.3828210569202504 | 28.949227456839488 |4861429.717089311 |237892032.56328017|
| 14 |0.02075704418575049 | 161815.60121281142 |556661120245.1433 |896441767889219.8 |
* fitting plots of training and test error:
* fitting plots of degree 5 leave-one-out error:
* fitting plots of degree 5 five-fold error:
* fitting plots of degree 10 leave-one-out error:
* fitting plots of deg 10 five-fold error:
* fitting plots of degree 14 leave-one-out error:
* fitting plots of degree 14 five-fold error:
# part B 討論:
在此題中基本上與a大題大同小異，值得探討的是當degree越高，我們可以觀察到她的training error越低，也就是越接近over fitting，從上圖可以得知，錯誤率越低不見得會是我們需要的曲線權重，他可能會為了迎合某個點，而造出了非常誇張的彎曲點，之後在E大題會對DEGREE 14 的多項式進行正規化，可以觀察這些曲線的變化。
---
# part C 題目與作法:
(c) Generate data using y = sin(2πx) +ε with the noise ε∼ N(0, 0.04)
and (equal spacing) x ∈ [0, 1]. Show the fitting plots of the training error,
cross-validation errors for both leave-one-out and five-fold, and testing
errors via polynomial regression with degree 5, 10 and 14.
C的做法幾乎與B完全相同，基本上就只有差在給的資料集不同而已，就不詳細說明步驟了
* **主要程式:**
```python=
def partC():
x, y = dataGenC()
x_train, x_test = partial(x)
print(x_train)
y_train, y_test = partial(y)
print(y_train)
#將x,y帶入
W_poly = POLYNOMIAL_regression(x_train, y_train, 5)
print("deg5_poly_regression: y = ")
for i in range(6):
print(str(W_poly[i]) + "x^" + str(5 - i))
print("----------------train_err------------------")
y_train_pred = Poly_func(x_train,W_poly )
mse_err = mse_error(y_train_pred, y_train)
print(mse_err)
print("----------------cross_LOO_err------------------")
print(cross_Kfold_Poly_err(x, y,20,5))
print("----------------cross_five-fold_err------------------")
print(cross_Kfold_Poly_err(x, y,5,5))
print("----------------test_err------------------")
y_test_pred = Poly_func(x_test,W_poly )
mse_err = mse_error(y_test_pred, y_test)
print(mse_err)
W_poly = POLYNOMIAL_regression(x_train, y_train, 10)
print("deg10_poly_regression: y = ")
for i in range(11):
print(str(W_poly[i]) + "x^" + str(10 - i))
print("----------------train_err------------------")
y_train_pred = Poly_func(x_train,W_poly )
mse_err = mse_error(y_train_pred, y_train)
print(mse_err)
print("----------------cross_LOO_err------------------")
print(cross_Kfold_Poly_err(x, y,20,10))
print("----------------cross_five-fold_err------------------")
print(cross_Kfold_Poly_err(x, y,5,10))
print("----------------test_err------------------")
y_test_pred = Poly_func(x_test,W_poly )
mse_err = mse_error(y_test_pred, y_test)
print(mse_err)
W_poly = POLYNOMIAL_regression(x_train, y_train, 14)
print("deg14_poly_regression: y = ")
for i in range(15):
print(str(W_poly[i]) + "x^"+str(14-i))
print("----------------train_err------------------")
y_train_pred = Poly_func(x_train,W_poly )
mse_err = mse_error(y_train_pred, y_train)
print(mse_err)
print("----------------cross_LOO_err------------------")
print(cross_Kfold_Poly_err(x, y,20,14))
print("----------------cross_five-fold_err------------------")
print(cross_Kfold_Poly_err(x, y,5,14))
print("----------------test_err------------------")
y_test_pred = Poly_func(x_test,W_poly )
mse_err = mse_error(y_test_pred, y_test)
print(mse_err)
plt.title("C poly_regression")
plt.xlabel("x axis caption")
plt.ylabel("y axis caption")
plt.ylim(-4, 4)
plt.scatter(x_train, y_train, label="train-Data")
plt.scatter(x_test, y_test, label="test-Data")
W_poly = POLYNOMIAL_regression(x_train, y_train, 5)
plt.plot(x, Poly_func(x, W_poly), label='deg5_poly_regression')
W_poly = POLYNOMIAL_regression(x_train, y_train, 10)
plt.plot(x,Poly_func(x,W_poly ),label='deg10_poly_regression')
W_poly = POLYNOMIAL_regression(x_train, y_train, 14)
plt.plot(x,Poly_func(x,W_poly ),label='deg14_poly_regression')
plt.legend()
plt.show()
return
```
* **生產資料:**
```python=
def dataGenC():
line = np.linspace(0, 1, 20)
noise = np.random.normal(0, 0.04, line.shape)
testdata = np.arange(20,dtype='Float64')
for i in range(20):
testdata[i] =np.sin(2*np.pi* line[i])+ noise[i]
plt.title("origin data set")
plt.xlabel("x axis caption")
plt.ylabel("y axis caption")
plt.plot(line, testdata, '.')
plt.show()
return line, testdata
```
# part C 結果:
| degree | training error | leave-one-out error | five-fold error | testing error |
| ------ | -------------- | ------------------- | --------------- | ------------- |
| 5 |0.0011095718290334245 | 0.0016489605142597681 |0.09551009859631585 |0.22227555379058428|
| 10 |0.000858199725011139 | 0.02265465226492868 |2807.294257460612 |150430.99306095642|
| 14 |0.02075704418575049 | 161815.60121281142 |556661120245.1433 |896441767889219.8 |
* fitting plots of training and test error:
* fitting plots of degree 5 leave-one-out error:
* fitting plots of degree 5 five-fold error:
* fitting plots of degree 10 leave-one-out error:
* fitting plots of deg 10 five-fold error:
* fitting plots of degree 14 leave-one-out error:
* fitting plots of degree 14 five-fold error:
# part C 討論 :
在C的結果中，可以清楚的觀察到，對於這種資料分布LOOCV的錯誤率明顯比5TH-FOLD還要低上非常多，因為訓練資料與測試資料分隔的方式，將訓練出來的回歸線避免掉了太極端的狀況，而越低的DEGREE我們可以看到越容易接近理想的走向。
---
# part D 題目與作法:
(d)Consider the model in (b) with degree 14 via varying the number
training data points m, say, m = 60, 160, 320.
Show the five-fold crossvalidation errors,
testing error and the fitting plots with 75% for training and 25% for testing.
在D大題中，基本的做法也與B大題差不多，只差在資料量變多，題目要求也去掉了訓練、LOOCV的ERROR，而且只需要驗證DEGREE 14的多項式回歸。
* **主要程式:**
```python=
def partD():
x, y = dataGenD(60)
x_train, x_test = partial(x,45)
print(x_train)
y_train, y_test = partial(y,45)
print(y_train)
W_poly = POLYNOMIAL_regression(x_train, y_train, 14)
print("deg14_poly_regression: y = ")
for i in range(15):
print(str(W_poly[i]) + "x^"+str(14-i))
print("----------------cross_five-fold_err------------------")
print(cross_Kfold_Poly_err(x, y,5,14))
print("----------------test_err------------------")
y_test_pred = Poly_func(x_test,W_poly )
mse_err = mse_error(y_test_pred, y_test)
print(mse_err)
#將x,y帶入
plt.title("Data M=60 Poly regression D")
plt.xlabel("x axis caption")
plt.ylabel("y axis caption")
plt.ylim(-15, 15)
plt.scatter(x_train, y_train, label="train-Data")
plt.scatter(x_test, y_test, label="test-Data")
plt.plot(x,Poly_func(x,W_poly ),label='deg14_poly_regression')
plt.legend()
plt.show()
x, y = dataGenD(160)
x_train, x_test = partial(x,120)
print(x_train)
y_train, y_test = partial(y,120)
print(y_train)
#將x,y帶入
W_poly = POLYNOMIAL_regression(x_train, y_train, 14)
print("deg14_poly_regression: y = ")
for i in range(15):
print(str(W_poly[i]) + "x^"+str(14-i))
print("----------------cross_five-fold_err------------------")
print(cross_Kfold_Poly_err(x, y,5,14))
print("----------------test_err------------------")
y_test_pred = Poly_func(x_test,W_poly )
mse_err = mse_error(y_test_pred, y_test)
print(mse_err)
plt.title("Data M=160 Poly regression D")
plt.xlabel("x axis caption")
plt.ylabel("y axis caption")
plt.ylim(-15, 15)
plt.scatter(x_train, y_train, label="train-Data")
plt.scatter(x_test, y_test, label="test-Data")
plt.plot(x,Poly_func(x,W_poly ),label='deg14_poly_regression')
plt.legend()
plt.show()
x, y = dataGenD(320)
x_train, x_test = partial(x,240)
print(x_train)
y_train, y_test = partial(y,240)
print(y_train)
W_poly = POLYNOMIAL_regression(x_train, y_train, 14)
print("deg14_poly_regression: y = ")
for i in range(15):
print(str(W_poly[i]) + "x^" + str(14 - i))
print("----------------cross_five-fold_err------------------")
print(cross_Kfold_Poly_err(x, y,5,14))
print("----------------test_err------------------")
y_test_pred = Poly_func(x_test,W_poly )
mse_err = mse_error(y_test_pred, y_test)
print(mse_err)
#將x,y帶入
plt.title("Data M=320 Poly regression D")
plt.xlabel("x axis caption")
plt.ylabel("y axis caption")
plt.ylim(-15, 15)
plt.scatter(x_train, y_train, label="train-Data")
plt.scatter(x_test, y_test, label="test-Data")
plt.plot(x,Poly_func(x,W_poly ),label='deg14_poly_regression')
plt.legend()
plt.show()
return
```
* **生產資料:**
```python=
def dataGenD(m):
line = np.linspace(-3, 3, m)
noise = np.random.normal(0, 1, line.shape)
testdata = 2 * line + noise
plt.title("origin data set")
plt.xlabel("x axis caption")
plt.ylabel("y axis caption")
plt.plot(line, testdata, '.')
plt.show()
return line, testdata
```
# part D 結果:
| data points | five-fold error | testing error |
| ----------- | ----------------- | ------------------ |
| 60 |298365216.1927856 |151039266575.15793 |
| 160 |397793952.5387165 |13587221376.92273 |
| 320 |51117609.60851878 |228455691.36662143 |
* data points =60 fitting plots with 75% for training and 25% for testing.
* data points =160 fitting plots with 75% for training and 25% for testing.
* data points =320 fitting plots with 75% for training and 25% for testing.
# part D 討論:
在D中，我們可以觀察到不同資料量下回歸線的擬合程度，一樣都是degree 14 的多項式，資料量變大之後 overfitting 的程度就有明顯改善。
---
# part E 題目與作法:
(e) Consider again the model in (b) with degree 14 via regularization:
Compare the results derived by setting λ = 0, 0.001/m , 1/m, 1000/m ,
where m = 20 is the number of data points (with x = 0, 1/(m−1) , 2/(m−1), . . . , 1).
Show the five-fold cross-validation errors, testing errors and
the fitting plots with regularization using the following equation:
1. 在E題中，基本作法完全套用B大題，主要不同的地方在於要改動回歸線權重產出的部分，使其能受到λ的影響，進行一定程度的正規化。
2. 新的資料點分布現在落在0~1之間，我對產出資料做了修改。
3. 在回歸函式中，我新增並修改以下程式碼片段，使其計算出新的權重
```python=
I = np.identity(np.matmul(x.T, x).shape[0])
W_poly_reg = np.matmul(np.matmul(np.linalg.inv(np.matmul(x.T, x)+(λ*I)), x.T), y)
return W_poly_reg
```
4. 交叉驗證也需要一定程度的介面修改
* **主要程式:**
```python=
def partE(λ):
x, y = dataGenE()
x_train, x_test = partial(x)
y_train, y_test = partial(y)
#將x,y帶入
plt.title("λ="+str(λ)+ " poly_regression")
plt.xlabel("x axis caption")
plt.ylabel("y axis caption")
plt.ylim(-10, 10)
plt.scatter(x_train, y_train, label="train-Data")
plt.scatter(x_test, y_test, label="test-Data")
W_poly = W_poly_regularization(x_train, y_train, 14,λ)
print("deg14_poly_regression: y = ")
for i in range(15):
print(str(W_poly[i]) + "x^"+str(14-i))
print("----------------train_err------------------")
y_train_pred = Poly_func(x_train,W_poly )
mse_err = mse_error(y_train_pred, y_train)
print(mse_err)
print("----------------cross_LOO_err------------------")
print(cross_Kfold_Poly_REG_err(x, y,20,14,λ))
print("----------------cross_five-fold_err------------------")
print(cross_Kfold_Poly_REG_err(x, y,5,14,λ))
print("----------------test_err------------------")
y_test_pred = Poly_func(x_test,W_poly )
mse_err = mse_error(y_test_pred, y_test)
print(mse_err)
p = np.poly1d(W_poly.flatten())
plt.plot(x,p(x),label='deg14_poly_regression')
plt.legend()
plt.show()
return
```
* **生產資料:**
```python=
def dataGenE():
line = np.linspace(0, 1, 20)
noise = np.random.normal(0, 1, line.shape)
testdata = 2 * line + noise
plt.title("origin data set")
plt.xlabel("x axis caption")
plt.ylabel("y axis caption")
plt.plot(line, testdata, '.')
plt.show()
return line,testdata
```
* **正規化回歸函式**
```python=
def W_poly_regularization(x, y, k,λ):
ori_x=x[:, np.newaxis]
x = np.concatenate((x[:, np.newaxis], np.ones((x.shape[0], 1))), axis=1)
for i in range(2, k+1):
left_col = ori_x ** (i)
x = np.concatenate((left_col, x), axis=1)
y = y[:, np.newaxis]
#print(y)
I = np.identity(np.matmul(x.T, x).shape[0])
W_poly_reg = np.matmul(np.matmul(np.linalg.inv(np.matmul(x.T, x)+(λ*I)), x.T), y)
return W_poly_reg
```
* **交叉驗證**
```python=
def cross_Kfold_Poly_REG_err(x, y, k,deg,λ):
Krate = 1 / k
startptr = 0
mse_err=0
for _ in range(k):
x_train, x_test = partial(x,startptr, rate=Krate)
y_train, y_test = partial(y,startptr, rate=Krate)
W_poly = W_poly_regularization(x_train, y_train, deg,λ)
y_test_pred = Poly_func(x_test, W_poly)
K_err = mse_error(y_test_pred, y_test)
startptr = startptr + math.floor(x.size * Krate)
mse_err = mse_err + K_err
mse_err=mse_err/k
return mse_err
```
# part E 結果:
| λ | training error | leave-one-out error | five-fold error | testing error |
| ------- | ------------------ | ------------------- | ------------------- |:------------------ |
| 0 | 0.1533886181858655 | 1763.5185783622564 | 230.99616778478216 | 57848173767.33128 |
| 0.00005 | 0.506509927535406 | 0.4828003286343222 | 0.45396984017017655 | 644.0181792641503 |
| 0.05 | 0.8093246791329191 | 0.7749077434083728 | 0.6780822033924777 | 40.72248732864744 |
| 50 | 1.8309932739925157 | 1.5754631598282187 | 1.6315905800419586 | 2.5862747960589214 |
* λ=0 fitting plots with regularization using the following equation:
* λ=0.00005 fitting plots with regularization using the following equation:
* λ=0.05 fitting plots with regularization using the following equation:
* λ=50 fitting plots with regularization using the following equation:
# part E 討論:
在此題的結果下，我們可以觀察到λ為0時不影響原始回歸線，而透過λ的提升有助於降DEGREE的現象，當λ大到某個值時，此多項式幾乎已趨近一個常數多項式了。
Sign in with Wallet
Connect another wallet