# pytorch基本知識
## 張量
- 張量是一個陣列，也是儲存數值集合的資料結構，其中的數值可以用索引(index)來讀取
```python
import torch # <1>
a = torch.ones(3) # <2>創建一個三維的一軸向量，其元素皆為1
a[1]
```
### 創建座標
- 創建二軸張量
- torch.Size([3, 2])代表第0軸有3維，第1軸有2維
```python
points = torch.tensor([[4.0, 1.0], [5.0, 3.0], [2.0, 1.0]])
```
```python
points = torch.zeros(3, 2)
```
### 利用索引值操作張量
```python
points = torch.tensor([[4.0, 1.0], [5.0, 3.0], [2.0, 1.0]])
point[1:, :] #前面是列索引，後面是行索引
```
### 管理張量dtype屬性
- 張量常用型別為float32, float64
- 用dtype =
```python
double_points = torch.ones(10, 2, dtype = torch.double)
double_points.dtype
```
### 其他張量功能
- 轉置
```python
a = torch.ones(3, 2)
a_t = torch.transpose(a, 0, 1) #將a的第0軸跟第1軸進行轉置
some_t = torch.ones(3, 4, 5)
transpose_t = some_t.transpose(0, 2) #第0軸跟第2軸進行轉置
```
### 連續張量
- 在pytorch中，有一些操作只能在連續張量中進行，例如view()，利用contiguous()轉為連續張量
```python
points.is_contiguous() #True
points_t.is_contiguous() #False
points = torch.tensor([[4.0, 1.0], [5.0, 3.0], [2.0, 1.0]])
points_t = points.t()
points_t_cont = points_t.contiguous() #轉成連續張量
```
## 現實資料應用
### 載入一張圖片檔
```python
import numpy as np
import torch
import imageio #對不同類型圖片有統一api
img_arr = imageio.imread(r'C:\Users\User\mypython\dlwpt-code-master\data\p1ch4\image-dog\bobby.jpg')
img_arr.shape
```
- (720, 1280, 3):由左至右為空間大小、彩色通道數，img_arr為一個3軸張量
#### pytorch要求圖片資料張量為C(彩色通道) * H(高度) * W(寬度)
```python
img = torch.from_numpy(img_arr) #先將img轉成pytorch張量
out = img.permute(2, 0, 1) #將第2軸移至第0軸，原本的第0軸和第1軸移到後面
out.shape
```
- torch.Size([3, 720, 1280]
### 載入多張圖片
- 需要在第0軸插入批次軸(batch，即資料集中的圖片數量)
- 產生一個N(batch) * C(彩色通道) * H(高度) * W(寬度)
```python
batch_size = 100 #設定一次可載入100張圖片
batch = torch.zeros(batch_size, 3, 256, 256, dtype=torch.uint8) #希望每一種顏色強度都用8位元整數顯示
```
- 載入多張圖片
```python
import os
data_dir = r'C:\Users\User\mypython\dlwpt-code-master\data\p1ch4\image-cats'
filenames = [name for name in os.listdir(data_dir) if os.path.splitext(name)[-1] == '.png']
for i, filename in enumerate(filenames):
img_arr = imageio.imread(os.path.join(data_dir, filename))
img_t = torch.from_numpy(img_arr)
img_t = img_t.permute(2, 0, 1)
img_t = img_t[:3]
batch[i] = img_t
```
- os.path.splitext(name)[-1] == '.png':副檔名是否為png
- for i, filename in enumerate(filenames):可以一次取得同时获得元素的索引和值
- os.path.join(data_dir, filename):合併路徑
- img_t = img_t.permute(2, 0, 1):將第2軸移至第0軸，原本第0軸和第1軸往後移
- img_t = img_t[:3]:有些圖片會有表示透明度的第3軸，我們不需要，故只保留前面3軸
### 將資料內的像素質做轉換
- 神經網路預設輸入為浮點數，當輸入資料在0~1之間時，神經網路有最好的訓練效能
- 因此在讀入圖片後，要將張量中的整數轉換浮點數，並對像素值進行正規化
#### 轉換浮點數
```python
batch = batch.float()
```
#### 正規化1:除255
- 單一像素值介於0~255之間，若同除255，可能正規化效果
```python
batch /= 255.0
```
#### 正規化2:標準化
```python
n_channels = batch.shape[1]
for c in range(n_channels):
mean = torch.mean(batch[:, c])
std = torch.std(batch[:, c])
batch[:, c] = (batch[:, c]-mean)/std
```
- n_channels = batch.shape[1]:取得彩色通道的數量
- for c in range(n_channels):依次走訪每一個通道
### 3D立體圖片資料
- 涉及到CT醫學成像應用，例如每張圖片對應身體的一個切面
#### 舉例:CT掃描樣本
```python
#載入資料
dir_path = r'C:\Users\User\mypython\dlwpt-code-master\data\p1ch4\volumetric-dicom\2-LUNG 3.0 B70f-04083'
vol_arr = imageio.volread(dir_path, 'DICOM')
vol_arr.shape
```
- (99, 512, 512)
- volread 是 imageio 库中的一個函式，用於讀取多個圖像檔案並將它們組合成一個連續的三維體積數據。
- 輸入參數：該函式接受一個檔案名稱列表作為輸入，表示要讀取的圖像檔案列表。這些圖像檔案通常表示了體積數據在不同切片上的連續切面
- (depth, height, width) 形狀的數組，其中 depth 表示切片的數量，height 和 width 表示每個切片的高度和寬度
- 因為省略了通道軸，因此資料中各軸排序和pytorch要求不一致(N * C * H * W)
#### 使用unsqueeze()增加一個通道軸
```python
vol = torch.from_numpy(vol_arr).float()
vol = torch.unsqueeze(vol, 1) #在第一軸插入通道軸，維度為1
vol = torch.unsqueeze(vol, 2) #在第二軸插入深度，維度為1
```
- torch.Size([99, 1, 1, 512, 512])
### 表格資料
#### 載入葡萄酒資料
- pandas是最節省記憶體和時間，但pytorch和numpy互通性較佳，所以用numpy開csv
```python
import csv
wine_path = r'C:\Users\User\mypython\dlwpt-code-master\data\p1ch4\tabular-wine\winequality-white.csv'
wineq_numpy = np.loadtxt(wine_path, dtype = np.float32, delimiter = ";", skiprow = 1)
```
- np.loadtxt() 是NumPy庫中的函式，用於從文本檔案中讀取數據並將其轉換為NumPy數組
- skiprow=1 表示跳過第一行，這是因為數據檔案的第一行可能包含標題或其他不需要讀取的內容
#### 取得標題
```python
col_list = next(csv.reader(open(wine_path), delimiter = ";"))
```
#### 把numpy陣列轉為pytorch張量
```python
wineq = torch.from_numpy(wineq_numpy)
wineq.shape, wineq.dtype
```
- (torch.Size([4898, 12]), torch.float32)
#### 把目標張量抓出
```python
data = wineq[:, :-1]
target = wineq[:, -1]
target = wineq[:, -1].long()
```
#### 將目標張量進行one hot編碼
- 建立shape為[4898, 10]，10的原因為酒品值有1~10分
```python
target_onehot = torch.zeros(target.shape[0], 10)
a = target_onehot.scatter_(1, target.unsqueeze(1), 1.0)
a, a.shape
```
- a = target_onehot.scatter_(1, target.unsqueeze(1), 1.0):
1. 第一个参数 1 表示在第 1 维度上进行填充操作（这里是列维度）
2. 第二个参数 target.unsqueeze(1) :因為target_onehot有兩個軸(4898, 10)，所以要用unsqueeze(1)將target增加一軸(變成4898, 1)，使其軸數與target_onehot相同
3. 第三个参数 1.0 是要填充的值，表示对应位置上的元素置为 1.0
#### data的資料分析
```python
data_mean = torch.mean(data, dim=0)
data_var = torch.var(data, dim=0)
data_normalized = (data-data_mean)/torch.sqrt(data_var)
#辨別葡萄酒品質
#辨識葡萄酒品質，這邊定義品質小於3就歸為品質差
bad_indexes = target<=3
bad_indexes.shape, bad_indexes.dtype, bad_indexes.sum()
bad_data = data[bad_indexes]
#將資料分為好中壞，再來計算不同等級的葡萄酒，各個化學平均值是多少
mid_data = data[(target > 3) & (target <7)]
good_data = data[(target >=7)]
bad_mean = torch.mean(bad_data, dim=0)
mid_mean = torch.mean(mid_data, dim=0)
good_mean = torch.mean(good_data, dim=0)
for i, args in enumerate(zip(col_list, bad_mean, mid_mean, good_mean)):
print('{:2} {:20} {:6.2f} {:6.2f} {:6.2f}'.format(i, *args))
0 fixed acidity 7.60 6.89 6.73
1 volatile acidity 0.33 0.28 0.27
2 citric acid 0.34 0.34 0.33
3 residual sugar 6.39 6.71 5.26
4 chlorides 0.05 0.05 0.04
5 free sulfur dioxide 53.33 35.42 34.55
6 total sulfur dioxide 170.60 141.83 125.25
7 density 0.99 0.99 0.99
8 pH 3.19 3.18 3.22
9 sulphates 0.47 0.49 0.50
10 alcohol 10.34 10.26 11.42
```
```python
#設定一個總二氧化碳的閾值
total_sulfur_threshold = 141.83
total_sulfur_data = data[:, 6]
predicted_indexes = torch.lt(total_sulfur_data, total_sulfur_threshold)
predicted_indexes.shape, predicted_indexes.dtype, predicted_indexes.sum()
```
- torch.lt(total_sulfur_data(A), total_sulfur_threshold(B)):可以挑出A中，值小於B的項目索引
```python
actual_indexes = target > 5
actual_indexes.shape, actual_indexes.dtype, actual_indexes.sum()
(torch.Size([4898]), torch.bool, tensor(3258))
n_matches = torch.sum(actual_indexes & predicted_indexes).item()
n_predicted = torch.sum(predicted_indexes).item()
n_actual = torch.sum(actual_indexes).item()
n_matches, n_matches/n_predicted, n_matches/n_actual
(2018, 0.74000733406674, 0.6193984039287906)
```
- torch.sum(actual_indexes & predicted_indexes).item()
- torch.sum() 函数用于计算张量中所有元素的和
- 在PyTorch中，张量（Tensor）通常是在GPU上进行计算的，因此其结果仍然以张量的形式存储在GPU内存中。如果我们希望将结果作为标量值（scalar value）进行使用或进一步处理，需要使用.item()方法将其从张量中提取出来
### 時間序列:資料集(華盛頓特區共享自行車資料集)
- 該資料集包含2011~2012每小時的自行車租借術以及對應天氣和溫度等
- 時間序列資料具有訊姓
- 第一個目標是將2軸的資料轉為3軸的資料(根據天數切割成不同2軸資料集，再把這些2軸資料集合再一起，最終產生一個3軸資料集)
```python
bikes_numpy = npp.loadtxt(r'C:\Users\User\mypython\dlwpt-code-master\data\p1ch4\bike-sharing-dataset\hour-fixed.csv',dtype = np.float32, delimiter = ',', skiprow = 1, converters = {1:lamba x : float(x[8:10])})
bikes = torch.from_numpy(bikes_numpy)
tensor([[1.0000e+00, 1.0000e+00, 1.0000e+00, ..., 3.0000e+00, 1.3000e+01,
1.6000e+01],
[2.0000e+00, 1.0000e+00, 1.0000e+00, ..., 8.0000e+00, 3.2000e+01,
4.0000e+01],
[3.0000e+00, 1.0000e+00, 1.0000e+00, ..., 5.0000e+00, 2.7000e+01,
3.2000e+01],
...,
[1.7377e+04, 3.1000e+01, 1.0000e+00, ..., 7.0000e+00, 8.3000e+01,
9.0000e+01],
[1.7378e+04, 3.1000e+01, 1.0000e+00, ..., 1.3000e+01, 4.8000e+01,
6.1000e+01],
[1.7379e+04, 3.1000e+01, 1.0000e+00, ..., 1.2000e+01, 3.7000e+01,
4.9000e+01]])
```
- converters = {1:lamba x : float(x[8:10])}:代表將資料第一行(日期)轉換成單純1~31號(只取x的8~10:float(x[8:10]))
- 重塑資料集:資料原本型態為[17520, 17]，代表17520個小時資料，17代表每一個小時包含資料
1. 使其變成一個3軸資料集
```python
daily_bikes = bikes.view(-1, 24, bikes.shape[1])
daily_bikes.shape
#torch.Size([730, 24, 17])
daily_bikes = daily_bikes.transpose(1, 2)
daily_bikes.shape
#torch.Size([730, 17, 24])
```
- daily_bikes = bikes.view(-1, 24, bikes.shape[1]):view第一個參數為-1，代表其數字要自動計算，第1軸為24，代表一天24小時，view第一個參數為-1意為給定第1軸和第2軸的前提下，讓電腦自動算出第0軸的維度
#### 對天氣欄位進行one_hot編碼
- 先建立一個零張量，來存放one_hot編碼的結果，第0軸和第2軸有著與daily_bikes同樣的維度，但第1軸的值為4
```python
daily_weather_onehot = torch.zeros(daily_bikes.shape[0], 4, daily_bikes.shape[2])
#torch.Size([730, 4, 24])
daily_weather_onehot.scatter_(1, daily_bikes[:, 9, :].long().unsqueeze(1)-1, 1.0)
daily_bikes = torch.cat((daily_bikes, daily_weather_onehot), dim=1)
```
- daily_weather_onehot.scatter_(1, daily_bikes[:, 9, :].long().unsqueeze(1)-1, 1.0):將one_hot編碼的結果存到張量的第1軸中，這個操作是值皆在張量中進行，不會回傳新的張量
- torch.cat((daily_bikes, daily_weather_onehot), dim=1):沿著第1軸把資料串起來，将 daily_bikes 和 daily_weather_onehot 这两个张量在列维度上进行拼接，生成一个拼接后的结果张量。需要注意的是，两个输入张量在拼接维度上的大小必须一致，除拼接维度外的其他维度必须匹配
#### 另一種作法:將天氣欄位投影到0~1之間
```python
daily_bikes[:, 9, :] = (daily_bikes[:, 9, :] - 1.0)/ 3.0
```
#### 將氣溫欄位投影到0~1之間
- 這種正規化對模型訓練很有幫助
```python
temp = daily_bikes[:, 10, :]
temp_min = torch.min(temp)
temp_max = torch.min(temp)
daily_bikes[:, 10, :] = (daily_bikes[:, 10, :] - temp_min)/(temp_max - temp_min)
```
### 表示文字資料(以傲慢與偏見內容為例)
- 從兩種角度處理字元
1. 字元
2. 單字
#### 字元:ASCII碼
```python
with open(r'C:\Users\User\mypython\dlwpt-code-master\data\p1ch4\jane-austen\1342-0.txt', encoding='utf8') as f:
text = f.read()
#將文本的不同列存入一個串列
lines = text.split('\n')
line = lines[200]
'“Impossible, Mr. Bennet, impossible, when I am not acquainted with him'
```
- 創建一個張量儲存該文字
```python
letter_t = torch.zeros(len(line), 128)
#torch.Size([70, 128])
for i, letter in enumerate(line.lower().strip()):
letter_index = ord(letter) if ord(letter) < 128 else 0
letter_t[i][letter_index] = 1
```
#### 單字
- 先定義clean_words(),他可以接受一段文字，去除其中標點符號並將所有文字轉為小寫
```python
def clean_words(input_str):
punctuation = '.,;:"!?”“_-' #定義一些常用標點符號
word_list = input_str.lower().split()
word_list = [word.strip(punctuation) for word in word_list]
return word_list
words_in_line = clean_word(line)
['impossible',
'mr',
'bennet',
'impossible',
'when',
'i',
'am',
'not',
'acquainted',
'with',
'him']
```
- word_list = input_str.lower().split():以空白字元進行切割
- [word.strip(punctuation) for word in word_list]:利用strip()去除文字前、後的標點符號
#### 將所有資料納入
```python
#處理所有資料text
word_list = sorted(set(clean_word(text)))
#將word_list中的單字編入字典
word2index_dixt = {word: i for (i, word) in enumerate(word_list)}
{'': 0,
'#1342]': 1,
'$5,000)': 2,
"'_she": 3,
"'after": 4,
"'ah": 5,
"'as-is'": 6,
"'bingley": 7,
"'had": 8,
"'having": 9,
"'i": 10,
"'keep": 11,
"'lady": 12,
"'lately": 13,
"'lydia": 14,
"'mr": 15,
"'my": 16,
"'oh": 17,
"'s": 18,
"'this": 19,
"'tis": 20,
"'violently": 21,
"'yes,'": 22,
"'you": 23,
'($1': 24,
...
'by': 996,
'bye': 997,
'cake': 998,
'calculate': 999,
...}
```
- word_list = sorted(set(clean_word(text))):
1. 使用 set() 函数将单词列表转换为一个集合，以去除其中的重复单词。集合是一种无序的数据结构，其中每个元素都是唯一的
2. sorted:依照字母順序排列的串列
#### 將每個單字經onehot編碼後併入張量中
```python
word_t = torch.zeros(len(words_in_line), len(word2index_dixt))
for i, word in enumerate(words_in_line):
word_index = word2index_dict[word]
word_t[i][word_index] = 1
print('{} {} {}'.format(i, word_index, word))
```
## 學習的機制
### 以溫度計單位為例
```python
import numpy as np
import torch
torch.set_printoptions(edgeitems=2, linewidth=75)
t_c = [0.5, 14.0, 15.0, 28.0, 11.0, 8.0, 3.0, -4.0, 6.0, 13.0, 21.0] #攝氏溫度
t_u = [35.7, 55.9, 58.2, 81.9, 56.3, 48.9, 33.9, 21.8, 48.4, 60.4, 68.4] #溫度計上對應到的讀數
t_c = torch.tensor(t_c)
t_u = torch.tensor(t_u)
```
#### 使用線性模型進行測試
```python
def model(t_u, w, b):
return w * t_u + b #假設線性模型
def loss_fn(t_p, t_c):
squared_diffs = (t_p - t_c)**2
return squared_diffs.mean() #損失函數
#計算導數
#計算導數
def dloss_fn(t_p, t_c):
dsq_diffs = 2 * (t_p - t_c)
return dsq_diffs/t_p.size(0)
def dmodel_dw(t_u, w, b):
return t_u
def dmodel_db(t_u, w, b):
return 1.0
def grad_fn(t_u, t_c, t_p, w, b):
dloss_dtp = dloss_fn(t_p, t_c)
dloss_dw = dloss_dtp * dmodel_dw(t_u, w, b)
dloss_db = dloss_dtp * dmodel_db(t_u, w, b)
return torch.stack([dloss_dw.sum(), dloss_db.sum()])
#訓練迴圈
def training_loop(n_epochs, learning_rate, params, t_u, t_c, print_params = True):
for epoch in range(1, n_epochs+1):
w, b = params
t_p = model(t_u, w, b)
loss = loss_fn(t_p, t_c)
grad = grad_fn(t_u, t_c, t_p, w, b)
params = params - learning_rate * grad
print('Epoch %d: Loss %f'%(epoch, float(loss)))
if (print_params):
print('\tParams: ', params)
print('\tGrad: ', grad)
return params
training_loop(n_epochs = 100,
learning_rate = 1e-2,
params = torch.tensor([1.0, 0.0]),
t_u = t_u,
t_c = t_c
)
#結果
Epoch 1: Loss 1763.884766
Params: tensor([-44.1730, -0.8260])
Grad: tensor([4517.2964, 82.6000])
Epoch 2: Loss 5802484.500000
Params: tensor([2568.4011, 45.1637])
Grad: tensor([-261257.4062, -4598.9702])
Epoch 3: Loss 19408029696.000000
Params: tensor([-148527.7344, -2616.3931])
Grad: tensor([15109614.0000, 266155.6875])
Epoch 4: Loss 64915905708032.000000
Params: tensor([8589999.0000, 151310.8906])
Grad: tensor([-8.7385e+08, -1.5393e+07])
Epoch 5: Loss 217130525461053440.000000
Params: tensor([-4.9680e+08, -8.7510e+06])
Grad: tensor([5.0539e+10, 8.9023e+08])
Epoch 6: Loss 726257583152928129024.000000
Params: tensor([2.8732e+10, 5.0610e+08])
Grad: tensor([-2.9229e+12, -5.1486e+10])
Epoch 7: Loss 2429183416467662896627712.000000
Params: tensor([-1.6617e+12, -2.9270e+10])
Grad: tensor([1.6904e+14, 2.9776e+12])
Epoch 8: Loss 8125122549611731432050262016.000000
Params: tensor([9.6102e+13, 1.6928e+12])
Grad: tensor([-9.7764e+15, -1.7221e+14])
Epoch 9: Loss 27176882120842590626938030653440.000000
...
Grad: tensor([nan, nan])
Epoch 100: Loss nan
Params: tensor([nan, nan])
Grad: tensor([nan, nan])
```
- 結果:第11次訓練時就暴增為無限大，代表參數單次跟新幅度太大，導致發散了
```python=
training_loop(n_epochs = 100,
learning_rate = 1e-4,
params = torch.tensor([1.0, 0.0]),
t_u = t_u,
t_c = t_c
)
#結果
Epoch 1: Loss 1763.884766
Params: tensor([ 0.5483, -0.0083])
Grad: tensor([4517.2964, 82.6000])
Epoch 2: Loss 323.090515
Params: tensor([ 0.3623, -0.0118])
Grad: tensor([1859.5493, 35.7843])
Epoch 3: Loss 78.929634
Params: tensor([ 0.2858, -0.0135])
Grad: tensor([765.4666, 16.5122])
Epoch 4: Loss 37.552845
Params: tensor([ 0.2543, -0.0143])
Grad: tensor([315.0790, 8.5787])
Epoch 5: Loss 30.540283
Params: tensor([ 0.2413, -0.0149])
Grad: tensor([129.6733, 5.3127])
Epoch 6: Loss 29.351154
Params: tensor([ 0.2360, -0.0153])
Grad: tensor([53.3495, 3.9682])
Epoch 7: Loss 29.148884
Params: tensor([ 0.2338, -0.0156])
Grad: tensor([21.9304, 3.4148])
Epoch 8: Loss 29.113848
Params: tensor([ 0.2329, -0.0159])
Grad: tensor([8.9964, 3.1869])
Epoch 9: Loss 29.107145
...
Grad: tensor([-0.0533, 3.0226])
Epoch 100: Loss 29.022667
Params: tensor([ 0.2327, -0.0438])
Grad: tensor([-0.0532, 3.0226])
```
- 還是有一個問題，因為每次參數跟新非常小，故損失下降的速度非常緩慢，甚至停滯不前，上面問題可以通過讓學習率隨著訓練次數改變來解決，後面會解決
- 由上面epoch1可知w跟b級數差大概50倍，在某一個情況下，對某一參數來說適當的學習率，對另一個參數可能過大
- 上面問題可以透過調整輸入資料來降低梯度之間的差異，例如正規化
```python=
t_un = 0.1 * t_u
```
### 自動計算梯度(autograd)
```python=
import torch.optim as optim
def training_loop(n_epochs, optimizer, params, t_u, t_c):
for epoch in range(1, n_epochs+1):
t_p = model(t_u, *params)
loss = loss_fn(t_p, t_c)
optimizer.zero_grad()
loss.backward()
optimizer.step()
if epoch % 500 == 0:
print('Epoch %d: Loss %f'%(epoch, float(loss)))
return params
params = torch.tensor([1.0, 0.0], requires_grad=True)
learning_rate = 1e-2
optimizer = optim.SGD([params], lr=learning_rate)
training_loop(n_epochs=5000,
optimizer=optimizer,
params=params,
t_u=t_un,
t_c=t_c
)
#結果
Epoch 500: Loss 7.860115
Epoch 1000: Loss 3.828538
Epoch 1500: Loss 3.092191
Epoch 2000: Loss 2.957698
Epoch 2500: Loss 2.933134
Epoch 3000: Loss 2.928648
Epoch 3500: Loss 2.927830
Epoch 4000: Loss 2.927679
Epoch 4500: Loss 2.927652
Epoch 5000: Loss 2.927647
tensor([ 5.3671, -17.3012], requires_grad=True)
```
#### 使用另外一個優化器:Adam，其學習率會隨著優化器自動調整
```python=
params = torch.tensor([1.0, 0.0], requires_grad=True)
learning_rate = 1e-1
optimizer = optim.Adam([params], lr=learning_rate)
training_loop(n_epochs=2000,
optimizer=optimizer,
params=params,
t_u=t_u,
t_c=t_c
)
```
#### 切割訓練驗證集
```python=
n_samples = t_u.shape[0]
n_val = int(0.2 * n_samples)
shuffled_indices = torch.randperm(n_samples)
train_indices = shuffled_indices[:-n_val]
val_indices = shuffled_indices[-n_val:]
train_t_u = t_u[train_indices]
train_t_c = t_c[train_indices]
val_t_u = t_u[val_indices]
val_t_c = t_c[val_indices]
train_t_un = train_t_u * 0.1
val_t_un = val_t_u *0.1
```
- shuffled_indices = torch.randperm(n_samples):產生隨機的索引值
- train_indices = shuffled_indices[:-n_val]:取前80%
- val_indices = shuffled_indices[-n_val:]:取後20%
```python=
def training_loop(n_epochs, optimizer, params, train_t_u, val_t_u, train_t_c, val_t_c):
for epoch in range(1, n_epochs+1):
train_t_p = model(train_t_u, *params)
train_loss = loss_fn(train_t_p, train_t_c)
val_t_p = model(val_t_u, *params)
val_loss = loss_fn(val_t_p, val_t_c)
optimizer.zero_grad()
train_loss.backward()
optimizer.step()
if epoch <= 3 or epoch % 500 == 0:
print(f'Epoch {epoch}, Training loss {train_loss.item():.4f},'f'Validation loss {val_loss.item():.4f}')
return params
params = torch.tensor([1.0, 0.0], requires_grad=True)
learning_rate = 1e-2
optimizer = optim.SGD([params], lr=learning_rate)
training_loop(n_epochs=3000,
optimizer=optimizer,
params=params,
train_t_u=train_t_un,
val_t_u=val_t_un,
train_t_c=train_t_c,
val_t_c=val_t_c
)
#結果
Epoch 1, Training loss 66.5811,Validation loss 142.3890
Epoch 2, Training loss 38.8626,Validation loss 64.0434
Epoch 3, Training loss 33.3475,Validation loss 39.4590
Epoch 500, Training loss 7.1454,Validation loss 9.1252
Epoch 1000, Training loss 3.5940,Validation loss 5.3110
Epoch 1500, Training loss 3.0942,Validation loss 4.1611
Epoch 2000, Training loss 3.0238,Validation loss 3.7693
Epoch 2500, Training loss 3.0139,Validation loss 3.6278
Epoch 3000, Training loss 3.0125,Validation loss 3.5756
tensor([ 5.1964, -16.7512], requires_grad=True)
```
## 使用神經網路擬合資料
### 選擇最佳的激活函數
- 激活函數必須是非線性的，因為多個線性變換仍為線性變換，故透過激活函數神經網路才能模擬更加複雜的函數行為
- 激活函數必須是可微分的，才能計算梯度，但如果只有幾個點無法微分是沒問題的(如Hardtanh或ReLU)
- 這些函數必須至少一個敏感區域，輸入值得細微變動會對輸出值有顯著影響，這一點對模型來說很重要
- 絕大多數激活函數還有以下特性
1. 當輸入接近負無限大時，輸出趨近某個下限值
2. 當輸入接近正無限大時，輸出趨近某個上限值
### pytorch的nn模組
```python=
import torch.nn as nn
linear_model = nn.Linear(1, 1)
linear_model(t_un_val)
linear_model.weight
linear_model.bias
```
- nn.Linear(1, 1):可接受三個參數，第一個為輸入特徵的數量、第二個為輸出特徵的數量、以及線性模型是否有偏值
#### 批次優化
- 這邊需要將資料組成批次的理由有很多，其中一個為最大化利用手上的運算資源
- 為了符合nn.linear的輸入格式，需要把1D張量轉為矩陣
```python=
t_c = [0.5, 14.0, 15.0, 28.0, 11.0, 8.0, 3.0, -4.0, 6.0, 13.0, 21.0]
t_u = [35.7, 55.9, 58.2, 81.9, 56.3, 48.9, 33.9, 21.8, 48.4, 60.4, 68.4]
t_c = torch.tensor(t_c).unsqueeze(1) # <1>
t_u = torch.tensor(t_u).unsqueeze(1) # <1>
#torch.Size([11, 1])
def training_loop(n_epochs, optimizer, model, loss_fn, t_u_train, t_u_val, t_c_train, t_c_val):
for epoch in range(1, n_epochs+1):
t_p_train = model(t_u_train)
loss_train = loss_fn(t_p_train, t_c_train)
t_p_val = model(t_u_val)
loss_val = loss_fn(t_p_val, t_c_val)
optimizer.zero_grad()
loss_train.backward()
optimizer.step()
if epoch == 1 or epoch % 1000 == 0:
print(f'Epoch {epoch}, Training loss {loss_train.item():.4f},'f'Validation loss {loss_val.item():.4f}')
linear_model = nn.Linear(1, 1)
optimizer = torch.optim.SGD(linear_model.parameters(), lr=1e-2)
training_loop(n_epochs=3000,
optimizer=optimizer,
model=linear_model,
loss_fn=nn.MSELoss(),
t_u_train=t_un_train,
t_u_val=t_un_val,
t_c_train=t_c_train,
t_c_val=t_c_val
)
print(linear_model.weight)
print(linear_model.bias)
```
- torch.optim.SGD(linear_model.parameters(), lr=1e-2):可以用linear_model.parameters()抓參數
- loss_fn=nn.MSELoss():用nn的損失函數即可，不用自己寫
### 進入正題:神經網路
- 最簡地的神經網路:先建一層線性模組，套用激活函數，最後再建另一層線性模組，通常第一層線性模組+激活函數稱為hidden layer(隱藏層)
```python=
seq_model = nn.Sequential(nn.Linear(1, 13),
nn.Tanh(),
nn.Linear(13, 1))
```
- nn.Sequential(nn.Linear(1, 13),nn.Tanh(),nn.Linear(13, 1)):指定模組中要有13個神經元，且輸出層的第0軸要跟上一層的第1軸相同，特徵數1會先被展開成13個隱藏特徵，最後經過激活函數後再以線性方式結合成單一輸出特徵
#### 替參數命名
```python=
from collections import OrderedDict
seq_model = nn.Sequential(OrderedDict([
('hidden_linear', nn.Linear(1, 8)),
('hidden_activation', nn.Tanh()),
('output_linear', nn.Linear(8, 1))
]))
#結果
Sequential(
(hidden_linear): Linear(in_features=1, out_features=8, bias=True)
(hidden_activation): Tanh()
(output_linear): Linear(in_features=8, out_features=1, bias=True)
)
```
```python=
optimizer = optim.SGD(seq_model.parameters(), lr=1e-3) # <1>
training_loop(
n_epochs = 5000,
optimizer = optimizer,
model = seq_model,
loss_fn = nn.MSELoss(),
t_u_train = t_un_train,
t_u_val = t_un_val,
t_c_train = t_c_train,
t_c_val = t_c_val)
print('output', seq_model(t_un_val))
print('answer', t_c_val)
print('hidden', seq_model.hidden_linear.weight.grad)
---------------------------------------------------------
Epoch 1, Training loss 182.9724, Validation loss 231.8708
Epoch 1000, Training loss 6.6642, Validation loss 3.7330
Epoch 2000, Training loss 5.1502, Validation loss 0.1406
Epoch 3000, Training loss 2.9653, Validation loss 1.0005
Epoch 4000, Training loss 2.2839, Validation loss 1.6580
Epoch 5000, Training loss 2.1141, Validation loss 2.0215
output tensor([[-1.9930],
[20.8729]], grad_fn=<AddmmBackward>)
answer tensor([[-4.],
[21.]])
hidden tensor([[ 0.0272],
[ 0.0139],
[ 0.1692],
[ 0.1735],
[-0.1697],
[ 0.1455],
[-0.0136],
[-0.0554]])
```
#### 畫圖
```python=
from matplotlib import pyplot as plt
t_range = torch.arange(20., 90.).unsqueeze(1)
fig = plt.figure(dpi=600)
plt.xlabel("Fahrenheit")
plt.ylabel("Celsius")
plt.plot(t_u.numpy(), t_c.numpy(), 'o')
plt.plot(t_range.numpy(), seq_model(0.1 * t_range).detach().numpy(), 'c-')
plt.plot(t_u.numpy(), seq_model(0.1 * t_u).detach().numpy(), 'kx')
```
- 為什麼要使用detach(seq_model(0.1 * t_u).detach().numpy())
1. 在PyTorch中，預設情況下，張量（Tensor）與計算圖（Computation Graph）相關聯。計算圖是PyTorch用於實現自動求導的機制，它記錄了張量的計算過程以及相關的操作
2. 有時候我們希望將某個張量從計算圖中分離出來，使其成為一個獨立的張量，不再與原始的計算圖相關聯。這時候就可以使用 .detach() 方法
3. 停止梯度傳播：在某些情況下，我們可能不希望梯度從一個張量傳播到另一個張量，或者不需要對該張量進行梯度計算。通過使用 .detach() 方法，可以停止梯度在分離的張量中的傳播，從而節省計算資源並提高效率
4. 與其他庫進行兼容：有些庫只支持使用NumPy數組而不支持PyTorch張量。通過使用 .detach().numpy()，可以將張量轉換為NumPy數組，以便與其他庫進行無縫的集成和操作
## 從圖片中學習
### 經典的資料集:CIFAR-10資料集
- 蒐集了60000張32 * 32的小型彩色RGB圖片，每張圖片都有一個整數標籤
- 下載資料
```python=
from torchvision import datasets
data_path = '../data-unversioned/p1ch7'
cifar10 = datasets.CIFAR10(data_path, train=True, download=True)
cifar10_val = datasets.CIFAR10(data_path, train=False, download=True)
```
- cifar10 = datasets.CIFAR10(data_path, train=True, download=True):第一個參數為指定下載資料的存放路徑，第二個參數決定下載訓練資料還是驗證資料，第三個參數代表第一參數所指定路徑找不到時，下載資料
#### 存取資料
```python=
class_names = ['airplane','automobile','bird','cat','deer',
'dog','frog','horse','ship','truck']
img, label = cifar10[99]
img, label, class_names[label]
```
- 先建立一個xlass_name的字典
- 存取資料集中索引為99的項目
- cifar10資料集其實是PIL類別的RGB圖片
```python=
from matplotlib import pyplot as plt
plt.imshow(img)
plt.show()
```
### 將資料轉換為張量且正規化
```python=
from torchvision import transforms
tensor_cifar10 = datasets.CIFAR10(data_path, train=True, download=False, transform=transforms.ToTensor())
```
- 為了正規化，必須先行計算平均數和標準差
```python=
imgs = torch.stack([img_t for img_t, _ in tensor_cifar10], dim = 3)
#(torch.Size([3, 32, 32]), torch.float32)
imgs.view(3, -1).mean(dim = 1)
#tensor([0.4914, 0.4822, 0.4465])
imgs.view(3, -1).std(dim = 1)
#tensor([0.2470, 0.2435, 0.2616])
```
- torch.stack([img_t for img_t, _ in tensor_cifar10], dim = 3):將所有圖片做堆疊至一個額外的軸上，以方便計算平均數和標準差
- imgs.view(3, -1).mean(dim = 1):會指定保留第0軸(通道軸)，其餘軸的維度則合併在第1軸，變成一個3 * 51200000的陣列
```python=
transformed_cifar10 = datasets.CIFAR10(data_path,
train=True,
download=False,
transform=transforms.Compose([
transforms.ToTensor(),
transforms.Normalize((0.4914, 0.4822, 0.4465),(0.2470, 0.2435, 0.2616))
]))
cifar10_val = datasets.CIFAR10(data_path,
train=False,
download=False,
transform=transforms.Compose([
transforms.ToTensor(),
transforms.Normalize((0.4914, 0.4822, 0.4465),(0.2470, 0.2435, 0.2616))
]))
```
- 這時圖片已經和原本不同了，因為這些RGB圖片會因為正規化而超出0.0到1.0的範圍，進而改變整個通道的數量值級，Matplotlib會把超出範圍的像素以黑色表示
### 區分鳥和飛機
```python=
label_map = {0:0, 2:1}
class_names = ['airplane', 'bird']
cifar2 = [(img, label_map[label]) for img, label in transformed_cifar10 if label in [0, 2]]
cifar2_val = [(img, label_map[label]) for img, label in cifar10_val if label in [0, 2]]
```
- label_map = {0:0, 2:1}:將飛機的類別標籤對應到0(原本為0)，小鳥的類別標籤對應到1(原本為2)
#### 建立全連接模型
- 將圖片轉為1D向量，當作輸入
```python=
import torch.nn as nn
n_out = 2
model = nn.Sequential(nn.Linear(3072, 512),
nn.Tanh(),
nn.Linear(512, n_out))
```
- model = nn.Sequential(nn.Linear(3072, 512),
nn.Tanh(),
nn.Linear(512, n_out)):先計算每個樣本有多少個特徵，從shape(32 * 32 * 3 = 3072
#### 將輸出改為機率
```python=
softmax = nn.Softmax(dim = 1)
```
- softmax = nn.Softmax(dim = 1):會沿著第1軸套用softmax函數
#### 分類任務中的損失
- 在這個任務中，我們想最大化的正確類別的機率值，期望概似率很低時，損失函數的輸出要很高，反之概似率提高時，損失函數則要降低
- 剛好有一種損失函數符合需求，即負對數概似率(Negative log likelihood, NLL)，NLL = -sum(log(out_i[c_i]))
- 這邊把softmax換成logsoftmax以便讓計算更加穩定
- 期對每一個樣本而言，順序如下:
1. 以前饋方式運行神經網路以獲得最後一層線性輸出
2. 對最後一層的輸出套用softmax，將其轉換成機率
3. 把對應到正確的機率值單獨取出來
4. 將上述概似率取對數後加上負號便成為損失
```python=
n_out = 2
model = nn.Sequential(nn.Linear(3072, 512),
nn.Tanh(),
nn.Linear(512, n_out),
nn.LogSoftmax(dim=1))
loss = nn.NLLLoss()
out = model(img.view(-1).unsqueeze(0))
loss(out, torch.tensor([label]))
#tensor(1.1657, grad_fn=<NllLossBackward0>)
```
- img.view(-1).unsqueeze(0):將輸入轉為適當維度，nn模組要求將資料組合為batch，而批次大小要指定在張量的第0軸
#### 正式訓練
```python=
import torch
import torch.nn as nn
import torch.optim as optim
model = nn.Sequential(
nn.Linear(3072, 512),
nn.Tanh(),
nn.Linear(512, 2),
nn.LogSoftmax(dim=1))
learning_rate = 1e-2
optimizer = optim.SGD(model.parameters(), lr=learning_rate)
loss_fn = nn.NLLLoss()
n_epochs = 100
for epoch in range(n_epochs):
for img, label in cifar2:
out = model(img.view(-1).unsqueeze(0))
loss = loss_fn(out, torch.tensor([label]))
optimizer.zero_grad()
loss.backward()
optimizer.step()
print("Epoch: %d, Loss: %f" % (epoch, float(loss)))
-----------------------------------------------------
Epoch: 0, Loss: 8.245017
Epoch: 1, Loss: 8.690164
Epoch: 2, Loss: 7.133948
Epoch: 3, Loss: 2.909860
Epoch: 4, Loss: 11.549189
Epoch: 5, Loss: 4.331132
Epoch: 6, Loss: 9.456457
Epoch: 7, Loss: 11.872355
Epoch: 8, Loss: 9.359488
Epoch: 9, Loss: 10.043870
Epoch: 10, Loss: 4.542089
Epoch: 11, Loss: 7.758786
```
- 上面的優化方法圍在大迴圈中加入另一個內部迴圈，以便讓模型美評估完一張圖片後便反向傳播一次
- 但是有一個問題:如果每次只優化一個樣本，降低損失的參數跟新方向有可能不正確，讓優化方向變來變去，導致學習過程不穩定
- 還一個更好的方法，就是在進入一個訓練周期時，先將所有樣本順序打亂，在評估少量樣本的梯度並進行優化，，此方法稱為隨機梯度下降法
#### 利用小批次中所有樣本的累積結果跟新模型
```python=
model = nn.Sequential(
nn.Linear(3072, 512),
nn.Tanh(),
nn.Linear(512, 2),
nn.LogSoftmax(dim=1))
learning_rate = 1e-2
optimizer = optim.SGD(model.parameters(), lr=learning_rate)
loss_fn = nn.NLLLoss()
n_epochs = 100
for epoch in range(n_epochs):
train_loader = torch.utils.data.Dataloader(cifar2, batch_size = 2, shuffle = True)
for imgs, labels in train_loader:
batch_size = imgs.shape[0]
outputs = model(imgs.view(batch_size, -1))
loss = loss_fn(outputs, labels)
optimizer.zero_grad()
loss.backward()
optimizer.step()
print("Epoch: %d, Loss: %f" % (epoch, float(loss)))
-----------------------------------------------------
Epoch: 0, Loss: 0.289170
Epoch: 1, Loss: 0.741474
Epoch: 2, Loss: 0.418546
Epoch: 3, Loss: 0.382951
Epoch: 4, Loss: 0.475265
Epoch: 5, Loss: 0.382178
Epoch: 6, Loss: 0.389627
Epoch: 7, Loss: 0.166453
Epoch: 8, Loss: 0.407214
Epoch: 9, Loss: 0.327904
Epoch: 10, Loss: 0.330175
Epoch: 11, Loss: 0.249890
Epoch: 12, Loss: 0.473498
Epoch: 13, Loss: 0.321211
Epoch: 14, Loss: 0.288500
Epoch: 15, Loss: 0.311383
Epoch: 16, Loss: 0.374049
Epoch: 17, Loss: 0.524892
Epoch: 18, Loss: 0.319869
Epoch: 19, Loss: 0.507166
Epoch: 20, Loss: 0.244206
Epoch: 21, Loss: 0.351508
Epoch: 22, Loss: 0.255436
Epoch: 23, Loss: 0.278363
Epoch: 24, Loss: 0.118590
...
Epoch: 96, Loss: 0.012195
Epoch: 97, Loss: 0.021364
Epoch: 98, Loss: 0.011878
Epoch: 99, Loss: 0.015008
```
#### 計算驗證集的準確率
```python=
val_loader = torch.utils.data.DataLoader(cifar2_val, batch_size=64, shuffle=False)
correct = 0
total = 0
with torch.no_grad():
for imgs, labels in val_loader:
batch_size = imgs.shape[0]
outputs = model(imgs.view(batch_size, -1))
outputs = model(imgs.view(batch_size, -1))
_, predicted = torch.max(outputs, dim=1)
total += labels.shape[0]
correct += int((predicted == labels).sum())
print('Accuracy: %f' %(correct/total))
#Accuracy: 0.823500
```
#### 增加模型的層數
```python=
model_2 = nn.Sequential(
nn.Linear(3072, 1024),
nn.Tanh(),
nn.Linear(1024, 512),
nn.Tanh(),
nn.Linear(512, 128),
nn.Tanh(),
nn.Linear(128, 2),
nn.LogSoftmax(dim=1))
```
#### 計算第一個模型的參數數量
```python=
numel_list = [p.numel() for p in model.parameters()]
sum(numel_list), numel_list
#(1574402, [1572864, 512, 1024, 2])
```
### 全連接模型的限制
- 因為輸入是1維圖片張量中的訊息，這些層會計算RGB圖檔中的每一個像素值與其他像素的線性組合，由於整張圖片被壓縮成了向量，故像素之間的位置資訊被忽略了
- 簡單來說，全連接神經網路不具備平移不便性，為解決上述問題，需要擴增資料集(在訓練中對訓練圖片進行隨機位移)，但需要付出相當的代價，神經網路的隱藏特徵設數必須足夠強大，以便學習來自所有圖片的訊息
- 這個問題的解答為:卷積層(convolutional layers)
## 卷積神經網路
- 了解卷積如何解析區域訊息來表達評儀不便性
- 透過拋棄處理圖片中所有像素，只求取某像素與其隔壁鄰居的加權總合，將距離目標像素一定距離以外的權重設為0
- 優點:
1. 可針對鄰近像速進行區域化處理
2. 維持平移不變性
3. 模型參數數量顯著減少
- 工具
1. nn.Conv1d(時間序列)
2. nn.Conv2d(圖片)
3. nn.Conv3d(體積或影片)
```python=
conv = nn.Conv2d(3, 16, kernel_size=3)
conv = nn.Conv2d(3, 16, kernel_size=(3, 3))
```
- 第一個參數為輸入通道數，第二個為輸出通道數，第三個為卷積核大小
- 卷積層權重張量為和:因為核大小為3 * 3，故張量會包含3 * 3的部分，在考慮輸入通道數(上例為3)，因為輸出通道為16，本例中張量為16 * 3 * 3 * 3
```python=
conv.weight.shape, conv.bias.shape
#(torch.Size([16, 3, 3, 3]), torch.Size([16]))
```
#### 填補(padding)邊界
- 當使用3 * 3卷積層時，目標張量必須上下左右都有才行
- 透過padding=1進行邊緣填補
```python=
conv = nn.Conv2d(3, 1, kernel_size=3, padding=1) # <1>
output = conv(img.unsqueeze(0))
img.unsqueeze(0).shape, output.shape
```
### 利用卷積偵測特徵
```python=
with torch.no_grad():
conv.bias.zero_()
with torch.no_grad():
conv.weight.fill_(1.0 / 9.0)
output = conv(img.unsqueeze(0))
plt.figure(figsize=(10, 4.8)) # bookskip
ax1 = plt.subplot(1, 2, 1) # bookskip
plt.title('output') # bookskip
plt.imshow(output[0, 0].detach(), cmap='gray')
plt.subplot(1, 2, 2, sharex=ax1, sharey=ax1) # bookskip
plt.imshow(img.mean(0), cmap='gray') # bookskip
plt.title('input') # bookskip
plt.savefig('Ch8_F4_PyTorch.png') # bookskip
plt.show()
```
- 透過卷積過濾器輸出了原始圖片的平滑化版本
- 輸出像素之間相關性高
### 探討深度和池化
- 雖然捨棄全連接層，但產生了另一個問題，就是整體的訊息無法捕捉
- 透過downsampling(降採樣)實現
1. 取輸入像素的平均值(average pooling)，早期相當常見，但現在越來越少用
2. 取輸入像素的平均值(max pooling)，目前使用率最高的方法
3. 採用特定步長的卷積處理
```python=
pool = nn.MaxPool2d(2)
output = pool(img.unsqueeze(0))
img.unsqueeze(0).shape, output.shape
#(torch.Size([1, 3, 32, 32]), torch.Size([1, 3, 16, 16]))
```
### 建立神經網路
```python=
model = nn.Sequential(
nn.Conv2d(3, 16, kernel_size=3, padding=1),
nn.Tanh(),
nn.MaxPool2d(2),
nn.Conv2d(16, 8, kernel_size=3, padding=1),
nn.Tanh(),
nn.MaxPool2d(2),
# ... <1>
nn.Linear(8 * 8 * 8, 32),
nn.Tanh(),
nn.Linear(32, 2))
)
```
- 第一層卷積層能將包含RGB三個通道的圖片轉成16個通道，進而產生16個能辨識鳥類和飛機的低階特徵，緊接著被Tanh激活函數處理，接著經歷MaxPool2d池化作用將32 * 32轉成16 * 16，再經歷一次卷積，輸出16 * 16的8通道圖片，後面一樣
- 但上面無法將卷積輸出8 * 8的8通道圖片，轉為1D向量
### 建立nn.Module的子類別
```python=
class Net(nn.Module):
def __init__(self):
super().__init__()
self.conv1 = nn.Conv2d(3, 16, kernel_size=3, padding=1)
self.act1 = nn.Tanh()
self.pool1 = nn.MaxPool2d(2)
self.conv2 = nn.Conv2d(16, 8, kernel_size=3, padding=1)
self.act2 = nn.Tanh()
self.pool2 = nn.MaxPool2d(2)
self.fc1 = nn.Linear(8 * 8 * 8, 32)
self.act3 = nn.Tanh()
self.fc2 = nn.Linear(32, 2)
def forward(self, x):
out = self.pool1(self.act1(self.conv1(x)))
out = self.pool2(self.act2(self.conv2(out)))
out = out.view(-1, 8 * 8 * 8) # <1>
out = self.act3(self.fc1(out))
out = self.fc2(out)
return out
```
- def forward(self, x):該函式可以將模組的輸入轉為輸出
- out = out.view(-1, 8 * 8 * 8):扁平化卷積模組的輸出，因為不知道樣本的數量，所以把批次維度數量設為-1
### 函數式API
- 因為有一些子模組(如nn.Tanh和MaxPool2d)沒有參數，故登錄他們沒效率
```python=
import torch.nn.functional as F
class Net(nn.Module):
def __init__(self):
super().__init__()
self.conv1 = nn.Conv2d(3, 16, kernel_size=3, padding=1)
self.conv2 = nn.Conv2d(16, 8, kernel_size=3, padding=1)
self.fc1 = nn.Linear(8 * 8 * 8, 32)
self.fc2 = nn.Linear(32, 2)
def forward(self, x):
out = F.max_pool2d(torch.tanh(self.conv1(x)), 2)
out = F.max_pool2d(torch.tanh(self.conv2(out)), 2)
out = out.view(-1, 8 * 8 * 8)
out = torch.tanh(self.fc1(out))
out = self.fc2(out)
return out
```
### 訓練卷積網路
```python=
import datetime # <1>
def training_loop(n_epochs, optimizer, model, loss_fn, train_loader):
for epoch in range(1, n_epochs + 1): # <2>
loss_train = 0.0
for imgs, labels in train_loader: # <3>
outputs = model(imgs) # <4>
loss = loss_fn(outputs, labels) # <5>
optimizer.zero_grad() # <6>
loss.backward() # <7>
optimizer.step() # <8>
loss_train += loss.item() # <9>將此次訓練週期中的所有損失加總起來，記得用item()將損失轉換為python的數字
if epoch == 1 or epoch % 10 == 0:
print('{} Epoch {}, Training loss {}'.format(
datetime.datetime.now(), epoch,
loss_train / len(train_loader))) # <10>每批次的平均損失
```
```python=
train_loader = torch.utils.data.DataLoader(cifar2, batch_size=64,
shuffle=True) # <1>
model = Net() # <2>
optimizer = optim.SGD(model.parameters(), lr=1e-2) # <3>
loss_fn = nn.CrossEntropyLoss() # <4>
training_loop( # <5>
n_epochs = 100,
optimizer = optimizer,
model = model,
loss_fn = loss_fn,
train_loader = train_loader,
)
--------------------------------------------
2023-06-24 11:36:44.279190 Epoch1, Training loss 0.5592965162863397
2023-06-24 11:37:11.562273 Epoch10, Training loss 0.3309898807364664
2023-06-24 11:37:48.613165 Epoch20, Training loss 0.2880635774534219
2023-06-24 11:38:19.403839 Epoch30, Training loss 0.26272512251024793
2023-06-24 11:38:55.923187 Epoch40, Training loss 0.23889244378657098
2023-06-24 11:39:29.631086 Epoch50, Training loss 0.2226175694328964
2023-06-24 11:40:05.956924 Epoch60, Training loss 0.20463970915716925
2023-06-24 11:40:41.707331 Epoch70, Training loss 0.18857246979027037
2023-06-24 11:41:13.533230 Epoch80, Training loss 0.172813002423496
2023-06-24 11:41:48.439891 Epoch90, Training loss 0.15800453679766624
2023-06-24 11:42:25.592584 Epoch100, Training loss 0.1443308568589247
```
#### 計算準確率
```python=
def validate(model, train_loader, val_loader):
for name, loader in [('train', train_loader), ('vsl', val_loader)]:
correct = 0
total = 0
with torch.no_grad():
for imgs, labels in loader:
outputs = model(imgs)
_, predicted = torch.max(outputs, dim=1)
total += labels.shape[0]
correct += int((predicted == labels).sum())
print('Accuracy {}: {:.2f}'.format(name, correct/total))
train_loader = torch.utils.data.DataLoader(cifar2, batch_size=64, shuffle=False)
val_loader = torch.utils.data.DataLoader(cifar2_val, batch_size=64, shuffle=False)
validate(model, train_loader, val_loader)
------------------------------------------------
Accuracy train: 0.95
Accuracy vsl: 0.89
```
- with torch.no_grad():由於不需要做參數跟新，因此不需要計算梯度
- int((predicted == labels).sum()):得到一個布林陣列，接著算該批次有多少是正確的
- 用int將整數張量轉為整數，跟item()效果相同
#### 儲存與匯入模型參數
- 儲存:
```python=
torch.save(model.state_dict(), data_path + 'birds_vs_airplanes.pt')
```
- 匯入:
```python=
loaded_model = Net() # <1>
loaded_model.load_state_dict(torch.load(data_path+'birds_vs_airplanes.pt'))
```
### 再GPU上訓練(執行時間更短)
- 通常會建立一個device的變數，決定是否可用
```python=
device = (torch.device('cuda') if torch.cuda.is_available()
else torch.device('cpu'))
print(f"Training on device {device}.")
import datetime
def training_loop(n_epochs, optimizer, model, loss_fn, train_loader):
for epoch in range(1, n_epochs + 1):
loss_train = 0.0
for imgs, labels in train_loader:
imgs = imgs.to(device=device) # <1>
labels = labels.to(device=device)
outputs = model(imgs)
loss = loss_fn(outputs, labels)
optimizer.zero_grad()
loss.backward()
optimizer.step()
loss_train += loss.item()
if epoch == 1 or epoch % 10 == 0:
print('{} Epoch {}, Training loss {}'.format(
datetime.datetime.now(), epoch,
loss_train / len(train_loader)))
def validate(model, train_loader, val_loader):
accdict = {}
for name, loader in [("train", train_loader), ("val", val_loader)]:
correct = 0
total = 0
with torch.no_grad():
for imgs, labels in loader:
imgs = imgs.to(device=device)
labels = labels.to(device=device)
outputs = model(imgs)
_, predicted = torch.max(outputs, dim=1) # <1>
total += labels.shape[0]
correct += int((predicted == labels).sum())
print("Accuracy {}: {:.2f}".format(name , correct / total))
accdict[name] = correct / total
return accdict
```
### 增加模型的記憶容量:寬度
```python=
class NetWidth(nn.Module):
def __init__(self):
super().__init__()
self.conv1 = nn.Conv2d(3, 32, kernel_size=3, padding=1)
self.conv2 = nn.Conv2d(32, 16, kernel_size=3, padding=1)
self.fc1 = nn.Linear(16*8*8, 32)
self.fc2 = nn.Linear(32, 2)
def forward(self, x):
out = F.max_pool2d(torch.tanh(self.conv1(x)), 2)
out = F.max_pool2d(torch.tanh(self.conv2(out)), 2)
out = out.view(-1, 16*8*8)
out = torch.tanh(self.fc1(out))
out = self.fc2(out)
return out
```
- 如果不想定義參數具體數字，可以這樣寫
```python=
class NetWidth(nn.Module):
def __init__(self, n_channel):
super().__init__()
self.n_channel = n_channel
self.conv1 = nn.Conv2d(3, n_channel, kernel_size=3, padding=1)
self.conv2 = nn.Conv2d(n_channel, n_channel//2, kernel_size=3, padding=1)
self.fc1 = nn.Linear((n_channel//2)*8*8, 32)
self.fc2 = nn.Linear(32, 2)
def forward(self, x):
out = F.max_pool2d(torch.tanh(self.conv1(x)), 2)
out = F.max_pool2d(torch.tanh(self.conv2(out)), 2)
out = out.view(-1, (self.n_channel//2)*8*8)
out = torch.tanh(self.fc1(out))
out = self.fc2(out)
return out
```
- 模型容量越大，越能應付輸入資料中的變異性，但相對的更容易過度配適
### 協助模型收斂與普適:常規化(Regularization)
- 第一種方法為在損失中加入逞罰項，該項會逞罰較大的權重值，該方法可以使損失值變化更加平滑，並降低配適個別樣本所引起的參數改變
```python=
def training_loop_l2reg(n_epochs, optimizer, model, loss_fn, train_loader):
for epoch in range(1, n_epochs+1):
loss_train = 0
for imgs, labels in train_loader:
outputs = model(imgs)
loss = loss_fn(outputs, labels)
l2_lambda = 0.001
l2_norm = sum(p.pow(2.0).sum() for p in model.parameters())
loss = loss+l2_lambda * l2_norm
optimizer.zero_grad()
loss.backward()
optimizer.step()
loss_train += loss.item()
if epoch == 1 or epoch % 10 == 0:
print('{} Epoch{}, Training loss {}'.format(datetime.datetime.now(), epoch, loss_train/len(train_loader)))
```
- 如果要改用L1常規化:sum(p.pow(2.0).sum() for p in model.parameters())改為sum(p.abs().sum() for p in model.parameters())
- 現在可以在pytorch的SGD優化器設定weight_decay參數，並可以在
### 避免過度依賴單一輸入資料:丟棄法(Dropout)
- 其中一種說法，此方法能在每回合訓練中，有效使模型的神經元拓鋪結構產生些許差異，進而避免造成在過度配適適時會發生的記憶效應
- 只要在非線性激活函數與下一層線性或卷積層之間加入nn.Dropout模組
```python=
class NetDropout(nn.Module):
def __init__(self, n_chans1=32):
super().__init__()
self.n_chans1 = n_chans1
self.conv1 = nn.Conv2d(3, n_chans1, kernel_size=3, padding=1)
self.conv1_dropout = nn.Dropout2d(p=0.4)
self.conv2 = nn.Conv2d(n_chans1, n_chans1//2, kernel_size=3, padding=1)
self.conv2_dropout = nn.Dropout2d(p=0.4)
self.fc1 = nn.Linear(8*8*n_chans1//2, 32)
self.fc2 = nn.Linear(32, 2)
def forward(self, x):
out = F.max_pool2d(torch.tanh(self.conv1(x)), 2)
out = self.conv1_dropout(out)
out = F.max_pool2d(torch.tanh(self.conv2(out)), 2)
out = self.conv2_dropout(out)
out = out.view(-1, 8*8*self.n_chans1//2)
out = torch.tanh(self.fc1(out))
out = self.fc2(out)
return out
```
- 正常情況丟棄法只在訓練時期運作
- train模式:
```python=
model.train()
```
- val模式:
```python=
model.eval()
```
### 約束激活函數:批次正規化(Batch Normalization)
- 主要原理是:重新調整輸入資料的數值分布，以避免小批次資料落入激活函數的飽和區域(梯度變的不明顯，進而拖慢訓練的速度)
- 技術上來說，批次正規化會根據小批次樣本的平均值和標準差來平移模型中間層的輸入，並調整這些輸入的規模，持續根據小批次資料的統計資訊來正規化
```python=
class NetBatchNorm(nn.Module):
def __init__(self, n_chans1=32):
super().__init__()
self.n_chans1 = n_chans1
self.conv1 = nn.Conv2d(3, n_chans1, kernel_size=3, padding=1)
self.conv1_batchnorm = nn.BatchNorm2d(num_features=n_chans1)
self.conv2 = nn.Conv2d(n_chans1, n_chans1//2, kernel_size=3, padding=1)
self.conv2_batchnorm = nn.BatchNorm2d(num_features=n_chans1//2)
self.fc1 = nn.Linear(8*8*n_chans1//2, 32)
self.fc2 = nn.Linear(32, 2)
def forward(self, x):
out = self.conv1_batchnorm(self.conv1(x))
out = F.max_pool2d(torch.tanh(out), 2)
out = self.conv2_batchnorm(self.conv2(out))
out = F.max_pool2d(torch.tanh(out), 2)
out = out.view(-1, 8*8*self.n_chans1//2)
out = torch.tanh(self.fc1(out))
out = self.fc2(out)
return out
```
- 其目標在於改變輸入激活函數數值
- 和丟棄法一樣，批次正規化在訓練和評估的表現不同，在評估時，必須避免批次的影響，例如樣本A，B在同一個批次，應與不在同一個批次時的預測結果相同，因此在評估階段所進行的正規化，所使用的平均數和標準差必須一致(最好的方法為使用所有批次的平均值和標準差)
- 當呼叫model.eval()時，模型的批次正規化會直接使用所有批次資料的統計值來進行正規化，如果要繼續訓練，則只要要呼叫model.train()即可
### 深入探索更複雜的結構:深度
#### 跳躍連接(skip connections)
```python=
class NetRes(nn.Module):
def __init__(self, n_chans1=32):
super().__init__()
self.n_chans1 = n_chans1
self.conv1 = nn.Conv2d(3, n_chans1, kernel_size=3, padding=1)
self.conv2 = nn.Conv2d(n_chans1, n_chans1//2, kernel_size=3, padding=1)
self.conv3 = nn.Conv2d(n_chans1//2, n_chans1//2, kernel_size=3, padding=1)
self.fc1 = nn.Linear(4*4*n_chans1//2, 32)
self.fc2 = nn.Linear(32, 2)
def forward(self, x):
out = F.max_pool2d(torch.relu(self.conv1(x)), 2)
out = F.max_pool2d(torch.relu(self.conv2(out)), 2)
out1 = out
out = F.max_pool2d(torch.relu(self.conv3(out)) + out1, 2)
out = out.view(-1, 4*4*self.n_chans1//2)
out = torch.relu(self.fc1(out))
out = self.fc2(out)
return out
```
- 將第二卷積層跟第三卷積層的輸出相加，以上過程又稱恆等映射
- 從反向傳播的角度思考，在深層網路設置一個或多個較遠距離的跳躍連接，相當在遠距離的神經層之間多建立了一條直達通道，如此一來在進行遠距傳播時，可以減緩因進行大量導數及乘積操作所造成的問題，進而更直接的傳遞部分損失梯度到較遠的神經層
#### 利用pytorch建立非常深層的模型
```python=
class ResBlock(nn.Module):
def __init__(self, n_chans):
super(ResBlock, self).__init__()
self.conv = nn.Conv2d(n_chans, n_chans, kernel_size=3,
padding=1, bias=False) # <1>
self.batch_norm = nn.BatchNorm2d(num_features=n_chans)
torch.nn.init.kaiming_normal_(self.conv.weight,
nonlinearity='relu') # <2>
torch.nn.init.constant_(self.batch_norm.weight, 0.5)
torch.nn.init.zeros_(self.batch_norm.bias)
def forward(self, x):
out = self.conv(x)
out = self.batch_norm(out)
out = torch.relu(out)
return out + x
class NetResDeep(nn.Module):
def __init__(self, n_chans1=32, n_blocks=10):
super().__init__()
self.n_chans1 = n_chans1
self.conv1 = nn.Conv2d(3, n_chans1, kernel_size=3, padding=1)
self.resblocks = nn.Sequential(
*(n_blocks * [ResBlock(n_chans=n_chans1)]))
self.fc1 = nn.Linear(8 * 8 * n_chans1, 32)
self.fc2 = nn.Linear(32, 2)
def forward(self, x):
out = F.max_pool2d(torch.relu(self.conv1(x)), 2)
out = self.resblocks(out)
out = F.max_pool2d(out, 2)
out = out.view(-1, 8 * 8 * self.n_chans1)
out = torch.relu(self.fc1(out))
out = self.fc2(out)
return out
model = NetResDeep(n_chans1=32, n_blocks=100).to(device=device)
optimizer = optim.SGD(model.parameters(), lr=3e-3)
loss_fn = nn.CrossEntropyLoss()
training_loop(
n_epochs = 100,
optimizer = optimizer,
model = model,
loss_fn = loss_fn,
train_loader = train_loader,
)
all_acc_dict["res deep"] = validate(model, train_loader, val_loader)
```
Sign in with Wallet
Connect another wallet