---
# System prepended metadata

title: CCU_2020ML HW01 多項式回歸練習

---

# CCU_2020ML HW01 多項式回歸練習


>author: 楊淨
>date: 2020/04/07


    4105931 機器學習
    Assignment #1 – Regression
    Due Date : 04/07 23:59pm
    本次作業主要是實作 Unit-5、p.14 頁的迴歸公式，就可以完成(a) -(d)題；
    實作 Unit-7、p.11 頁的正規化公式，可以完成題(e)。
    本次作業需要自己實作上述公式，請勿使用任何現成的 regression 函式。**

    Use the linear model y = 2x +ε with zero-mean Gaussian noise ε∼ N(0,1)
    to  generate 20 data points with (equal spacing) x ∈ [−3, 3].


# part A 題目與作法:

    (a) Perform linear regression. 20 data points are split into 15 training
    samples and 5 testing samples (75% for training and 25% for testing).
    Show the fitting plots of the training error, cross-validation errors
    for both leave-one-out and five-fold, and testing errors.

1. 先將資料以`dataGen()`產出，並以x跟y接收後，再將他們以`partial()`分割成前75%訓練用資料及後25%測試用資料。
2. 接著再回歸方程式中利用x_train這筆訓練用資料`linear_regression(x_train,y_train)` 算出回歸線的各項權重，代回線性方程AX+B得出Y值
3. 利用`mse_error(y_train_pred, y_train)`算出誤差分數，便可以獲得訓練資料集中的mse_error 作為題目要的 training error。
4. 再利用寫好的`cross_Kfold_err(x, y,k)`可以自動將資料拆分成k等分進行Kfold驗證，分別訓練與驗證，算出其MSE_ERROR做為cross-validation errors，以此題來說，因為總資料量剛好是20，所以會拆分成20份19個訓練資料及1個測試資料即為LOOCV。
5. 同上利用`cross_Kfold_err(x, y,k)`，k代入5即為five-fold error
6. 最後利用拆分的test資料集 `y_test_pred = Fitted_func(x_test, W_lin[0], W_lin[1])` `mse_err = mse_error(y_test_pred, y_test)`計算出test error
7. 利用matplotlib庫繪製fitting plots

* **主要程式:**
```python=
def partA():
    x, y = dataGen()
    x_train,x_test = partial(x)
    y_train,y_test = partial(y)
    #將x,y帶入
    W_lin =linear_regression(x_train,y_train)
    print("linear_regression: y = " + str(W_lin[0]) + "x + " + str(W_lin[1]))
    print("----------------train_err------------------")
    y_train_pred = Fitted_func(x_train, W_lin[0], W_lin[1])
    mse_err = mse_error(y_train_pred, y_train)
    print(mse_err)
    print("----------------cross_LOO_err------------------")
    print(cross_Kfold_err(x, y,20))
    print("----------------cross_five-fold_err------------------")
    print(cross_Kfold_err(x, y,5))
    print("----------------test_err------------------")
    y_test_pred = Fitted_func(x_test, W_lin[0], W_lin[1])
    mse_err = mse_error(y_test_pred, y_test)
    print(mse_err)
    plt.title("train data set")
    plt.xlabel("x axis caption")
    plt.ylabel("y axis caption")
    plt.scatter(x_train, y_train, label="train-Data")
    plt.scatter(x_test, y_test, label="test-Data")
    plt.plot(x,Fitted_func(x, W_lin[0], W_lin[1]),label='Fitted function')
    plt.legend()
    plt.show()
    return
```
* **生產資料**
```python=
def dataGen():
    line = np.linspace(-3, 3, 20)
    noise = np.random.normal(0, 1, line.shape)
    testdata = 2 * line + noise
    print(line)
    print(testdata)
    plt.title("origin data set")
    plt.xlabel("x axis caption")
    plt.ylabel("y axis caption")
    plt.plot(line, testdata, '.')
    plt.show()
    return line,testdata
```
* **線性回歸**
```python=
def linear_regression(x, y):
    x = np.concatenate(( x[:, np.newaxis],np.ones((x.shape[0], 1))), axis=1)
    y = y[:, np.newaxis]
    w_lin = np.matmul(np.matmul(np.linalg.inv(np.matmul(x.T, x)), x.T), y)
    return w_lin
```
* **切分資料**
```python=
def partial(arr,start=15, rate=0.25):
    k=math.floor(arr.size * rate)
    test = arr[start: start + k]
    pretrain=np.array([])
    if (start != 0):
        pretrain=arr[:start]
    posttrain = arr[start + k :]
    train=np.append(pretrain,posttrain)
    return train, test
```
* **Mse_Error計算**
```python=
def mse_error(y_pred,y):
    mse = np.mean((y_pred- y)** 2)
    return mse
```
* **線性方程計算**
```python=
def Fitted_func(x, a, b):
    return (a * x)+b
```
* **cross-validation K-fold計算**
```python=
def cross_Kfold_err(x, y, k):
    Krate = 1 / k
    startptr = 0
    mse_err = 0
    plt.title("cross_"+str(k)+"-fold_err")
    plt.xlabel("x axis caption")
    plt.ylabel("y axis caption")
    for _ in range(k):
        x_train, x_test = partial(x,startptr, rate=Krate)
        y_train, y_test = partial(y, startptr, rate=Krate)
        plt.scatter(x_train, y_train, label="train-Data")
        plt.scatter(x_test, y_test, label="test-Data")
        W_lin = linear_regression(x_train, y_train)
        y_test_pred = Fitted_func(x_test, W_lin[0], W_lin[1])
        K_err = mse_error(y_test_pred, y_test)
        startptr = startptr + math.floor(x.size * Krate)
        mse_err = mse_err + K_err
        plt.plot(x,Fitted_func(x, W_lin[0], W_lin[1]),label=str(_)+'-th')
    mse_err = mse_err / k
    plt.legend()
    plt.show()
    return mse_err
```
# part A 結果:

| training error | leave-one-out error| five-fold error| testing error|
| --------      | --------           | --------      |  --------   |
| 0.6056858377530092    | 1.4539140490809153     | 1.313419308133971     | 2.784714551445104 |


* **fitting plots of training and test error:**
![](https://i.imgur.com/OAo1VhX.png)

* **fitting plots of leave-one-out error:** 
![](https://i.imgur.com/pippxeD.png)

* **fitting plots of five-fold error:**
![](https://i.imgur.com/qCJB1ne.png)

# part A 討論

由於我分割資料時並非隨機散佈，而是全部擠一坨在尾端。
![](https://i.imgur.com/flxoPF3.png)

這比較接近預測未來走勢的回歸模型，在這種模型下，對於測試資料的error通常會特別大，因為沒有歷史資料的輔助，對於超出X範圍的點，他無法給予一個相關性較高的Y值，而交叉驗證的LOOCV、kTH_FOLD，會避免這類極端狀況的發生，所以壓低了ERROR值。

---

# part B 題目與作法:
    (b) Perform polynomial regression with degree 5, 10 and 14, respectively.
    For each case, show the fitting plots of the training error, 
    crossvalidation errors (both leave-one-out and five-fold) and 
    testing errors.


1. 做法與A基本上大同小異，主要是差在原本線性回歸改成了多項式回歸而已
2. 先將資料以`dataGen()`產出，並以x跟y接收後，再將他們以`partial()`分割成前75%訓練用資料及後25%測試用資料。
3. 接著再回歸方程式中利用x_train這筆訓練用資料`POLYNOMIAL_regression(x_train, y_train, 5)` 算出高次方回歸線的各項權重。
4. 利用`y_train_pred = Poly_func(x_train,W_poly )` `mse_error(y_train_pred, y_train)`算出誤差分數，便可以獲得訓練資料集中的mse_error 作為題目要的 training error。
5. 再利用寫好的`cross_Kfold_Poly_err(x, y,k,degree)`可以自動將資料拆分成k等分進行Kfold驗證，分別訓練與驗證，算出其MSE_ERROR做為cross-validation errors，以此題來說，因為總資料量剛好是20，所以會拆分成20份19個訓練資料及1個測試資料即為LOOCV。
6. 同上利用`cross_Kfold_Poly_err(x, y,k,degree)`，k代入5即為five-fold error
7. 最後用一開始拆分好的test資料集 `y_test_pred = Poly_func(x_test, W_lin[0], W_lin[1])` `mse_err = mse_error(y_test_pred, y_test)`計算出test error
8. 重複上述步驟將degree改成10、14
9. 利用matplotlib庫繪製fitting plots

* **主要程式:**
```python=
def partB():
    x, y = dataGen()
    x_train, x_test = partial(x)
    print(x_train)
    y_train, y_test = partial(y)
    print(y_train)
    #將x,y帶入
    W_poly = POLYNOMIAL_regression(x_train, y_train, 5)
    print("deg5_poly_regression: y = ")
    for i in range(6):
        print(str(W_poly[i]) + "x^" + str(5 - i))
    print("----------------train_err------------------")
    y_train_pred = Poly_func(x_train,W_poly )
    mse_err = mse_error(y_train_pred, y_train)
    print(mse_err)
    print("----------------cross_LOO_err------------------")
    print(cross_Kfold_Poly_err(x, y,20,5))
    print("----------------cross_five-fold_err------------------")
    print(cross_Kfold_Poly_err(x, y,5,5))
    print("----------------test_err------------------")
    y_test_pred = Poly_func(x_test,W_poly )
    mse_err = mse_error(y_test_pred, y_test)
    print(mse_err)

    W_poly = POLYNOMIAL_regression(x_train, y_train, 10)
    print("deg10_poly_regression: y = ")
    for i in range(11):
        print(str(W_poly[i]) + "x^"+str(10-i))
    print("----------------train_err------------------")
    y_train_pred = Poly_func(x_train,W_poly )
    mse_err = mse_error(y_train_pred, y_train)
    print(mse_err)
    print("----------------cross_LOO_err------------------")
    print(cross_Kfold_Poly_err(x, y,20,10))
    print("----------------cross_five-fold_err------------------")
    print(cross_Kfold_Poly_err(x, y,5,10))
    print("----------------test_err------------------")
    y_test_pred = Poly_func(x_test,W_poly )
    mse_err = mse_error(y_test_pred, y_test)
    print(mse_err)
    W_poly = POLYNOMIAL_regression(x_train, y_train, 14)
    print("deg14_poly_regression: y = ")
    for i in range(15):
        print(str(W_poly[i]) + "x^"+str(14-i))
    print("----------------train_err------------------")
    y_train_pred = Poly_func(x_train,W_poly )
    mse_err = mse_error(y_train_pred, y_train)
    print(mse_err)
    print("----------------cross_LOO_err------------------")
    print(cross_Kfold_Poly_err(x, y,20,14))
    print("----------------cross_five-fold_err------------------")
    print(cross_Kfold_Poly_err(x, y,5,14))
    print("----------------test_err------------------")
    y_test_pred = Poly_func(x_test,W_poly )
    mse_err = mse_error(y_test_pred, y_test)
    print(mse_err)
    
    plt.title("poly_regression")
    plt.xlabel("x axis caption")
    plt.ylabel("y axis caption")
    plt.ylim(-30, 30)
    plt.scatter(x_train, y_train, label="train-Data")
    plt.scatter(x_test, y_test, label="test-Data")
    W_poly = POLYNOMIAL_regression(x_train, y_train, 5)
    plt.plot(x, Poly_func(x, W_poly), label='deg5_poly_regression')
    W_poly = POLYNOMIAL_regression(x_train, y_train, 10)
    plt.plot(x,Poly_func(x,W_poly ),label='deg10_poly_regression')
    W_poly = POLYNOMIAL_regression(x_train, y_train, 14)
    plt.plot(x,Poly_func(x,W_poly ),label='deg14_poly_regression')
    plt.legend()
    plt.show()

    return

```
* **生產資料**
```python=
def dataGen():
    line = np.linspace(-3, 3, 20)
    noise = np.random.normal(0, 1, line.shape)
    testdata = 2 * line + noise
    print(line)
    print(testdata)
    plt.title("origin data set")
    plt.xlabel("x axis caption")
    plt.ylabel("y axis caption")
    plt.plot(line, testdata, '.')
    plt.show()
    return line,testdata
```
* **多項式回歸**
```python=
def POLYNOMIAL_regression(x, y, k):
    ori_x=x[:, np.newaxis]
    x = np.concatenate((x[:, np.newaxis], np.ones((x.shape[0], 1))), axis=1)
    for i in range(2, k+1):
        left_col = ori_x ** (i)
        x = np.concatenate((left_col, x), axis=1)
    y = y[:, np.newaxis]
    #print(y)
    w_POLY = np.matmul(np.matmul(np.linalg.inv(np.matmul(x.T, x)), x.T), y)
    return w_POLY
```
* **切分資料**
```python=
def partial(arr,start=15, rate=0.25):
    k=math.floor(arr.size * rate)
    test = arr[start: start + k]
    pretrain=np.array([])
    if (start != 0):
        pretrain=arr[:start]
    posttrain = arr[start + k :]
    train=np.append(pretrain,posttrain)
    return train, test
```
* **Mse_Error計算**
```python=
def mse_error(y_pred,y):
    mse = np.mean((y_pred- y)** 2)
    return mse
```
* **多項式方程計算**
```python=
def Poly_func(x, w):
    y=x*0
    for i in range(w.size):
        y = y + w[w.size-i-1] * (x ** i)
    return y
```
* **cross-validation K-fold計算**
```python=
def cross_Kfold_Poly_err(x, y, k,deg):
    Krate = 1 / k
    startptr = 0
    mse_err = 0
    plt.title("cross_"+str(k)+"-fold_POLY_err")
    plt.xlabel("x axis caption")
    plt.ylabel("y axis caption")
    plt.ylim(-10, 10)
    for _ in range(k):
        x_train, x_test = partial(x,startptr, rate=Krate)
        y_train, y_test = partial(y,startptr, rate=Krate)
        W_poly = POLYNOMIAL_regression(x_train, y_train, deg)
        y_test_pred = Poly_func(x_test, W_poly)
        K_err = mse_error(y_test_pred, y_test)
        startptr = startptr + math.floor(x.size * Krate)
        mse_err = mse_err + K_err
        plt.scatter(x_train, y_train, label="train-Data")
        plt.scatter(x_test, y_test, label="test-Data")
        plt.plot(x, Poly_func(x, W_poly), label=str(_)+'TH deg-'+str(deg)+'_poly_regression')
    plt.legend()
    plt.show()
    mse_err=mse_err/k
    return mse_err
```

# part B 結果:

| degree | training error | leave-one-out error | five-fold error | testing error |
| ------ | -------------- | ------------------- | --------------- | ------------- |
|   5    |0.7159022520706798  | 1.6834928238066844 |187.44561217194774 |211.6211352155488|
|   10   |0.3828210569202504  | 28.949227456839488 |4861429.717089311  |237892032.56328017|
|   14   |0.02075704418575049 | 161815.60121281142 |556661120245.1433  |896441767889219.8 |


* fitting plots of training and test error:
 ![](https://i.imgur.com/8cM1d2w.png)

* fitting plots of degree 5 leave-one-out error:
 ![](https://i.imgur.com/4aUor3K.png)

* fitting plots of degree 5 five-fold error:
 ![](https://i.imgur.com/4Uxoa89.png)

* fitting plots of degree 10 leave-one-out error:
 ![](https://i.imgur.com/Y9cSx4s.png)

* fitting plots of deg 10 five-fold error:
 ![](https://i.imgur.com/UtoCFpg.png)

* fitting plots of degree 14 leave-one-out error:
 ![](https://i.imgur.com/IJJ8uIk.png)

* fitting plots of degree 14 five-fold error:
 ![](https://i.imgur.com/ClTzeYQ.png)
 
# part B 討論:

在此題中基本上與a大題大同小異，值得探討的是當degree越高，我們可以觀察到她的training error越低，也就是越接近over fitting，從上圖可以得知，錯誤率越低不見得會是我們需要的曲線權重，他可能會為了迎合某個點，而造出了非常誇張的彎曲點，之後在E大題會對DEGREE 14 的多項式進行正規化，可以觀察這些曲線的變化。


---
# part C 題目與作法:
    (c) Generate data using y = sin(2πx) +ε with the noise ε∼ N(0, 0.04)
    and (equal spacing) x ∈ [0, 1]. Show the fitting plots of the training error,
    cross-validation errors for both leave-one-out and five-fold, and testing
    errors via polynomial regression with degree 5, 10 and 14.

C的做法幾乎與B完全相同，基本上就只有差在給的資料集不同而已，就不詳細說明步驟了

* **主要程式:**
```python=
def partC():
    x, y = dataGenC()
    x_train, x_test = partial(x)
    print(x_train)
    y_train, y_test = partial(y)
    print(y_train)
    #將x,y帶入
    W_poly = POLYNOMIAL_regression(x_train, y_train, 5)
    print("deg5_poly_regression: y = ")
    for i in range(6):
        print(str(W_poly[i]) + "x^" + str(5 - i))
        
    print("----------------train_err------------------")
    y_train_pred = Poly_func(x_train,W_poly )
    mse_err = mse_error(y_train_pred, y_train)
    print(mse_err)
    print("----------------cross_LOO_err------------------")
    print(cross_Kfold_Poly_err(x, y,20,5))
    print("----------------cross_five-fold_err------------------")
    print(cross_Kfold_Poly_err(x, y,5,5))
    print("----------------test_err------------------")
    y_test_pred = Poly_func(x_test,W_poly )
    mse_err = mse_error(y_test_pred, y_test)
    print(mse_err)

    W_poly = POLYNOMIAL_regression(x_train, y_train, 10)
    print("deg10_poly_regression: y = ")
    for i in range(11):
        print(str(W_poly[i]) + "x^" + str(10 - i))
        
    print("----------------train_err------------------")
    y_train_pred = Poly_func(x_train,W_poly )
    mse_err = mse_error(y_train_pred, y_train)
    print(mse_err)
    print("----------------cross_LOO_err------------------")
    print(cross_Kfold_Poly_err(x, y,20,10))
    print("----------------cross_five-fold_err------------------")
    print(cross_Kfold_Poly_err(x, y,5,10))
    print("----------------test_err------------------")
    y_test_pred = Poly_func(x_test,W_poly )
    mse_err = mse_error(y_test_pred, y_test)
    print(mse_err)
    W_poly = POLYNOMIAL_regression(x_train, y_train, 14)
    print("deg14_poly_regression: y = ")

    for i in range(15):
        print(str(W_poly[i]) + "x^"+str(14-i))

    print("----------------train_err------------------")
    y_train_pred = Poly_func(x_train,W_poly )
    mse_err = mse_error(y_train_pred, y_train)
    print(mse_err)
    print("----------------cross_LOO_err------------------")
    print(cross_Kfold_Poly_err(x, y,20,14))
    print("----------------cross_five-fold_err------------------")
    print(cross_Kfold_Poly_err(x, y,5,14))
    print("----------------test_err------------------")
    y_test_pred = Poly_func(x_test,W_poly )
    mse_err = mse_error(y_test_pred, y_test)
    print(mse_err)

    plt.title("C poly_regression")
    plt.xlabel("x axis caption")
    plt.ylabel("y axis caption")
    plt.ylim(-4, 4)
    plt.scatter(x_train, y_train, label="train-Data")
    plt.scatter(x_test, y_test, label="test-Data")
    W_poly = POLYNOMIAL_regression(x_train, y_train, 5)
    plt.plot(x, Poly_func(x, W_poly), label='deg5_poly_regression')
    W_poly = POLYNOMIAL_regression(x_train, y_train, 10)
    plt.plot(x,Poly_func(x,W_poly ),label='deg10_poly_regression')
    W_poly = POLYNOMIAL_regression(x_train, y_train, 14)
    plt.plot(x,Poly_func(x,W_poly ),label='deg14_poly_regression')
    plt.legend()
    plt.show()
    return
```
* **生產資料:**
```python=
def dataGenC():
    line = np.linspace(0, 1, 20)
    noise = np.random.normal(0, 0.04, line.shape)
    testdata = np.arange(20,dtype='Float64')
    for i in range(20):
        testdata[i] =np.sin(2*np.pi* line[i])+ noise[i]
    plt.title("origin data set")
    plt.xlabel("x axis caption")
    plt.ylabel("y axis caption")
    plt.plot(line, testdata, '.')
    plt.show()
    return line, testdata
```
# part C 結果:

| degree | training error | leave-one-out error | five-fold error | testing error |
| ------ | -------------- | ------------------- | --------------- | ------------- |
|   5    |0.0011095718290334245  | 0.0016489605142597681 |0.09551009859631585 |0.22227555379058428|
|   10   |0.000858199725011139  | 0.02265465226492868 |2807.294257460612  |150430.99306095642|
|   14   |0.02075704418575049 | 161815.60121281142 |556661120245.1433  |896441767889219.8 |

* fitting plots of training and test error:
 ![](https://i.imgur.com/3tJ9bIg.png)

* fitting plots of degree 5 leave-one-out error:
 ![](https://i.imgur.com/5Pj9pwJ.png)

* fitting plots of degree 5 five-fold error:
 ![](https://i.imgur.com/Xmavh5l.png)

* fitting plots of degree 10 leave-one-out error:
 ![](https://i.imgur.com/4qnsqFu.png)

* fitting plots of deg 10 five-fold error:
 ![](https://i.imgur.com/r5DSipg.png)

* fitting plots of degree 14 leave-one-out error:
 ![](https://i.imgur.com/Od1BTby.png)

* fitting plots of degree 14 five-fold error:
![](https://i.imgur.com/sfdezYx.png)

# part C 討論 :
在C的結果中，可以清楚的觀察到，對於這種資料分布LOOCV的錯誤率明顯比5TH-FOLD還要低上非常多，因為訓練資料與測試資料分隔的方式，將訓練出來的回歸線避免掉了太極端的狀況，而越低的DEGREE我們可以看到越容易接近理想的走向。


---
# part D 題目與作法:
    (d)Consider the model in (b) with degree 14 via varying the number
    training data points m, say, m = 60, 160, 320. 
    Show the five-fold crossvalidation errors, 
    testing error and the fitting plots with 75% for training and 25% for testing.

在D大題中，基本的做法也與B大題差不多，只差在資料量變多，題目要求也去掉了訓練、LOOCV的ERROR，而且只需要驗證DEGREE 14的多項式回歸。

* **主要程式:**
```python=
def partD():
    x, y = dataGenD(60)
    x_train, x_test = partial(x,45)
    print(x_train)
    y_train, y_test = partial(y,45)
    print(y_train)

    W_poly = POLYNOMIAL_regression(x_train, y_train, 14)
    print("deg14_poly_regression: y = ")

    for i in range(15):
        print(str(W_poly[i]) + "x^"+str(14-i))

    print("----------------cross_five-fold_err------------------")
    print(cross_Kfold_Poly_err(x, y,5,14))
    print("----------------test_err------------------")
    y_test_pred = Poly_func(x_test,W_poly )
    mse_err = mse_error(y_test_pred, y_test)
    print(mse_err)
    #將x,y帶入
    plt.title("Data M=60  Poly regression D")
    plt.xlabel("x axis caption")
    plt.ylabel("y axis caption")
    plt.ylim(-15, 15)
    plt.scatter(x_train, y_train, label="train-Data")
    plt.scatter(x_test, y_test, label="test-Data")

    plt.plot(x,Poly_func(x,W_poly ),label='deg14_poly_regression')
    plt.legend()
    plt.show()

    x, y = dataGenD(160)
    x_train, x_test = partial(x,120)
    print(x_train)
    y_train, y_test = partial(y,120)
    print(y_train)
    #將x,y帶入


    W_poly = POLYNOMIAL_regression(x_train, y_train, 14)
    print("deg14_poly_regression: y = ")

    for i in range(15):
        print(str(W_poly[i]) + "x^"+str(14-i))
    print("----------------cross_five-fold_err------------------")
    print(cross_Kfold_Poly_err(x, y,5,14))
    print("----------------test_err------------------")
    y_test_pred = Poly_func(x_test,W_poly )
    mse_err = mse_error(y_test_pred, y_test)
    print(mse_err)

    plt.title("Data M=160  Poly regression D")
    plt.xlabel("x axis caption")
    plt.ylabel("y axis caption")
    plt.ylim(-15, 15)
    plt.scatter(x_train, y_train, label="train-Data")
    plt.scatter(x_test, y_test, label="test-Data")
    plt.plot(x,Poly_func(x,W_poly ),label='deg14_poly_regression')
    plt.legend()
    plt.show()

    x, y = dataGenD(320)
    x_train, x_test = partial(x,240)
    print(x_train)
    y_train, y_test = partial(y,240)
    print(y_train)

    W_poly = POLYNOMIAL_regression(x_train, y_train, 14)
    print("deg14_poly_regression: y = ")

    for i in range(15):
        print(str(W_poly[i]) + "x^" + str(14 - i))

    print("----------------cross_five-fold_err------------------")
    print(cross_Kfold_Poly_err(x, y,5,14))
    print("----------------test_err------------------")
    y_test_pred = Poly_func(x_test,W_poly )
    mse_err = mse_error(y_test_pred, y_test)
    print(mse_err)
    #將x,y帶入

    plt.title("Data M=320  Poly regression D")
    plt.xlabel("x axis caption")
    plt.ylabel("y axis caption")
    plt.ylim(-15, 15)
    plt.scatter(x_train, y_train, label="train-Data")
    plt.scatter(x_test, y_test, label="test-Data")

    plt.plot(x,Poly_func(x,W_poly ),label='deg14_poly_regression')
    plt.legend()
    plt.show()
    return
```
* **生產資料:**
```python=
def dataGenD(m):
    line = np.linspace(-3, 3, m)
    noise = np.random.normal(0, 1, line.shape)
    testdata = 2 * line + noise
    plt.title("origin data set")
    plt.xlabel("x axis caption")
    plt.ylabel("y axis caption")
    plt.plot(line, testdata, '.')
    plt.show()
    return line, testdata
```
# part D 結果:



| data points | five-fold error   | testing error      |
| ----------- | ----------------- | ------------------ |
| 60          |298365216.1927856 |151039266575.15793 |
| 160         |397793952.5387165 |13587221376.92273  |
| 320         |51117609.60851878 |228455691.36662143 |
		
* data points =60 fitting plots with 75% for training and 25% for testing.
![](https://i.imgur.com/MEHuo1K.png)

* data points =160 fitting plots with 75% for training and 25% for testing.
![](https://i.imgur.com/zniYv09.png)

* data points =320 fitting plots with 75% for training and 25% for testing.
![](https://i.imgur.com/LdPO2D4.png)

# part D 討論:

在D中，我們可以觀察到不同資料量下回歸線的擬合程度，一樣都是degree 14 的多項式，資料量變大之後 overfitting 的程度就有明顯改善。

---
# part E 題目與作法:
    (e) Consider again the model in (b) with degree 14 via regularization: 
![](https://i.imgur.com/CaG1bDT.png)

    Compare the results derived by setting λ = 0, 0.001/m , 1/m, 1000/m , 
    where m = 20 is the number of data points (with x = 0, 1/(m−1) , 2/(m−1), . . . , 1).
    Show the five-fold cross-validation errors, testing errors and
    the fitting plots with regularization using the following equation:
![](https://i.imgur.com/5A7pZT1.png)

1. 在E題中，基本作法完全套用B大題，主要不同的地方在於要改動回歸線權重產出的部分，使其能受到λ的影響，進行一定程度的正規化。
2. 新的資料點分布現在落在0~1之間，我對產出資料做了修改。
3. 在回歸函式中，我新增並修改以下程式碼片段，使其計算出新的權重 
![](https://i.imgur.com/aJzdBoG.png)
```python=
    I = np.identity(np.matmul(x.T, x).shape[0])
    W_poly_reg = np.matmul(np.matmul(np.linalg.inv(np.matmul(x.T, x)+(λ*I)), x.T), y)
    return W_poly_reg
```


4. 交叉驗證也需要一定程度的介面修改


* **主要程式:**
```python=
def partE(λ):
    x, y = dataGenE()
    x_train, x_test = partial(x)
    y_train, y_test = partial(y)
    #將x,y帶入

    plt.title("λ="+str(λ)+ " poly_regression")
    plt.xlabel("x axis caption")
    plt.ylabel("y axis caption")
    plt.ylim(-10, 10)
    plt.scatter(x_train, y_train, label="train-Data")
    plt.scatter(x_test, y_test, label="test-Data")

    W_poly = W_poly_regularization(x_train, y_train, 14,λ)
    print("deg14_poly_regression: y = ")

    for i in range(15):
        print(str(W_poly[i]) + "x^"+str(14-i))

    print("----------------train_err------------------")
    y_train_pred = Poly_func(x_train,W_poly )
    mse_err = mse_error(y_train_pred, y_train)
    print(mse_err)
    print("----------------cross_LOO_err------------------")
    print(cross_Kfold_Poly_REG_err(x, y,20,14,λ))

    print("----------------cross_five-fold_err------------------")
    print(cross_Kfold_Poly_REG_err(x, y,5,14,λ))
    print("----------------test_err------------------")
    y_test_pred = Poly_func(x_test,W_poly )
    mse_err = mse_error(y_test_pred, y_test)
    print(mse_err)

    p = np.poly1d(W_poly.flatten())
    plt.plot(x,p(x),label='deg14_poly_regression')
    plt.legend()
    plt.show()
    return
```
* **生產資料:**
```python=
def dataGenE():
    line = np.linspace(0, 1, 20)
    noise = np.random.normal(0, 1, line.shape)
    testdata = 2 * line + noise
    plt.title("origin data set")
    plt.xlabel("x axis caption")
    plt.ylabel("y axis caption")
    plt.plot(line, testdata, '.')
    plt.show()
    return line,testdata
```
* **正規化回歸函式**
```python=
def W_poly_regularization(x, y, k,λ):
    ori_x=x[:, np.newaxis]
    x = np.concatenate((x[:, np.newaxis], np.ones((x.shape[0], 1))), axis=1)
    for i in range(2, k+1):
        left_col = ori_x ** (i)
        x = np.concatenate((left_col, x), axis=1)
    y = y[:, np.newaxis]
    #print(y)
    I = np.identity(np.matmul(x.T, x).shape[0])
    W_poly_reg = np.matmul(np.matmul(np.linalg.inv(np.matmul(x.T, x)+(λ*I)), x.T), y)
    return W_poly_reg
```
* **交叉驗證**
```python=
def cross_Kfold_Poly_REG_err(x, y, k,deg,λ):
    Krate = 1 / k
    startptr = 0
    mse_err=0
    for _ in range(k):
        x_train, x_test = partial(x,startptr, rate=Krate)
        y_train, y_test = partial(y,startptr, rate=Krate)
        W_poly = W_poly_regularization(x_train, y_train, deg,λ)
        y_test_pred = Poly_func(x_test, W_poly)
        K_err = mse_error(y_test_pred, y_test)
        startptr = startptr + math.floor(x.size * Krate)
        mse_err = mse_err + K_err
    mse_err=mse_err/k
    return mse_err
```
    
    
# part E 結果:

| λ       | training error     | leave-one-out error | five-fold error     | testing error      |
| ------- | ------------------ | ------------------- | ------------------- |:------------------ |
| 0       | 0.1533886181858655 | 1763.5185783622564  | 230.99616778478216  | 57848173767.33128  |
| 0.00005 | 0.506509927535406  | 0.4828003286343222  | 0.45396984017017655 | 644.0181792641503  |
| 0.05    | 0.8093246791329191 | 0.7749077434083728  | 0.6780822033924777  | 40.72248732864744  |
| 50      | 1.8309932739925157 | 1.5754631598282187  | 1.6315905800419586  | 2.5862747960589214 |

* λ=0 fitting plots with regularization using the following equation:
![](https://i.imgur.com/9rXTYZ9.png)
* λ=0.00005 fitting plots with regularization using the following equation:
![](https://i.imgur.com/pVKNCkL.png)
* λ=0.05 fitting plots with regularization using the following equation:
![](https://i.imgur.com/zjH4IJn.png)
* λ=50 fitting plots with regularization using the following equation:
![](https://i.imgur.com/yeRf6xb.png)
# part E 討論:
在此題的結果下，我們可以觀察到λ為0時不影響原始回歸線，而透過λ的提升有助於降DEGREE的現象，當λ大到某個值時，此多項式幾乎已趨近一個常數多項式了。