---
# System prepended metadata

title: PyTorch & TF tutorial

---

# PyTorch & TF tutorial
# Pytorch tutorial
* tensor和numpy array很像，不過不一樣的地方是tensor的運算可以藉由GPU來做平行化。
* 如何建立一個tensor物件
    ```python=
    import torch
    x=torch.tensor([1,2,3])
    # x: tensor([1, 2, 3])
    ```
    或是在建立時指定data type
    ```python=
    x=torch.tensor([1,2,3],dtype=torch.float32)
    #x: tensor([1., 2., 3.])
    ```
    **有時在training 的時候會遇到input data和model parameter一個是float32一個是float64，可以把下面tutorial的torch.tensor(...,dtype=torch.float64)改成float32，或是直接用torch.FloatTensor([1,2,3])最簡單。**
    
    也可以初始化成全部都0或1的tensor等等...
    ```python=
    x=torch.ones((2,4))
    #x: tensor([[1., 1., 1., 1.],
    #           [1., 1., 1., 1.]])
    
    x=torch.zeros((2,4))
    #x: tensor([[0., 0., 0., 0.],
    #           [0., 0., 0., 0.]])
    
    ```
* tensor的reshape
    ```python=
    x=torch.tensor([[1,2,3,4],[5,6,7,8]])
    #x: tensor([[1, 2, 3, 4],
    #           [5, 6, 7, 8]])
    
    x=x.view((4,-1))
    #tensor([[1, 2],
    #        [3, 4],
    #        [5, 6],
    #        [7, 8]])
    
    x=x.view(8)
    #tensor([1, 2, 3, 4, 5, 6, 7, 8])
    .
    .
    .
    ```
[torch.tensor的操作們](https://pytorch.org/docs/stable/torch.html)
* torch.tensor和np.array之間的轉換
    * torch.tensor to np.array
        ```python=
        a=torch.ones(5)
        b=a.numpy()
        #a: tensor([1., 1., 1., 1., 1.])
        #b: [1. 1. 1. 1. 1.]
        ```
    * np.array to torch.tensor
        ```python=
        a=np.ones(5)
        b=torch.from_numpy(a)
        #a: [1. 1. 1. 1. 1.]
        #b: tensor([1., 1., 1., 1., 1.],dtype=torch.float64)
        ```
    值得注意的是a和b其實是share同一個address，因此只要對a的數值做變動，b的數值也會跟著變。如果要讓他們獨立的話要用下面的方法。
    * torch.tensor to np.array
        ```python=
        a=torch.ones(5)
        b=np.array(a)
        ```
    * np.array to torch.tensor
        ```python=
        a=np.ones(5)
        b=torch.tensor(a)
        ```
* 將數值移到GPU上
    ```python=
    a=torch.tensor([1,2,3],device="cuda")
    #or
    a=torch.tensor([1,2,3])
    a=a.to(torch.device("cuda")) '''or''' a=a.cuda()
    
    # a: tensor([1, 2, 3], device='cuda:0')
    ```
* autograd
    pytorch提供自動在tensor上進行微分的功能。只要將tensor的requires_grad屬性設成true，pytorch就會開始記錄對於此tensor做的各種操作。然後最後只要呼叫tensor的`.backward()`，就可以把所有與此tensor有關的所有tensor的偏微分算出來，存在各tensor的`.grad`裡面。例如:
    ```python=
    x=torch.tensor([1,2,3],dtype=torch.float64,requires_grad=True)
    y=x+2
    #y: tensor([3., 4., 5.], device='cuda:0', 
    #          dtype=torch.float64,grad_fn=<AddBackward0>)
    z=y*y*3
    #z: tensor([27., 48., 75.], device='cuda:0', 
    #          dtype=torch.float64,grad_fn=<MulBackward0>)
    out=z.mean()
    #out: tensor(50.0000, device='cuda:0', dtype=torch.float64,
    #            grad_fn=<MeanBackward0>)
    out.backward()
    #d(out)/dx is stored in x.grad
    #x.grad: tensor([ 6.,  8., 10.], dtype=torch.float64)
    ```
    

    如果要讓pytorch結束對於tensor的紀錄(例如進入到testing state)，就call tensor的`.detach_()`，或是將code包在`with torch.no_grad():`裡面就好了，或是在function前面加上@torch.no_grad()
    ```python=
    x.detach_()
    #or
    with torch.no_grad:
        x+1
        ...
    #or
    @torch.no_grad()
    def my_function_no_grad():
        x+1
        ...
    ```
    值得注意的一點是：grad在反向傳播的過程中是會累加的，這代表每一次進行反向傳播，梯度都會累加到之前的梯度。所以反向傳播之前需把梯度清零。方法如下：
    ```python=
    # 以下底線結尾的函數是inplace操作，會修改自身的值，就像add_
    x.grad.data.zero_()
    ```
    
    
![](https://i.imgur.com/wk2ZYpl.png)
* 定義一個model
    在pytorch中如果要定義一個Net，可以直接繼承torch.nn.Module。然後在繼承以後有兩個member function一定要實做。```__init__```和```forward```，init部分就是在定義model的架構，forward部分就是在定義數值要怎麼在model裡面運算。
    ```python=
    import torch
    import torch.nn as nn
    import torch.nn.functional as F

    class Net(nn.Module):
        def __init__(self):
            super().__init__()
            self.conv1=nn.Conv2d(1,6,3)#1input,6output,3*3kernel
            self.conv2=nn.Conv2d(6,16,3)
            self.fc1=nn.Linear(16*6*6,120)                            
            self.fc2=nn.Linear(120,84)
            self.fc3=nn.Linear(84,10)
            
        def forward(self,x):
            x=F.max_pool2d(F.relu(self.conv1(x)),2)
            # 2 is for 2*2 max pooling window size
            x=F.max_pool2d(F.relu(self.conv2(x)),2)
            x=x.view(x.size(0),-1)
            x=F.relu(self.fc1(x))
            x=F.relu(self.fc2(x))
            x=self.fc3(x)
            return x

    ```
    或是可以用nn.Sequential簡化forward的部分
    ```python=
    class Net(nn.Module):
        def __init__(self):
            super().__init__()
            self.net=nn.Sequential(
                nn.Conv2d(1,6,3),
                nn.ReLU(),
                nn.MaxPool2d((2,2)),
                nn.Conv2d(6,16,3),
                nn.Relu(),
                nn.MaxPool2d((2,2)),
                nn.Flatten(),
                nn.Linear(16*6*6,120),
                nn.ReLU(),
                nn.Linear(120,84),
                nn.ReLU(),
                nn.Linear(84,10)
            )
        def forward(self,x):
            return self.net(x)
    ```
* load data
    只要用pytorch訓練模型基本上都會用到torch.utils.data.DataLoader來讀取訓練資料並傳給model，而這個DataLoader物件需要吃一個torch.utils.data.Dataset的物件當作它的data來源，因此先從torch.utils.data.Dataset物件介紹起。
    * torch.utils.data.Dataset
        pytorch定義的一個dataset虛擬物件，要在自己定義的dataset中繼承這個物件，並且一定要實作__getitem__和__len__這兩個function，前者是用來根據傳進來的index回傳dataset中的一組data(包含input和label)，後者是回傳dataset的size。例如
    ```python=
    from torch.utils.data import Dataset
    
    class my_dataset(Dataset):#繼承Dataset物件
        def __init__(self,...,...):
            super().__init__()
            self.Data=........
            self.Label=.......
            
        def __getitem__(self,index):
            #通常會在get item裡面實作data augmentation、transformation
            data=torch.tensor(self.Data[index])
            label=torch.tensor(self.Label[index])
            return data,label
            
        def __len__(self):
            return len(self.Label)
    data=my_dataset("PATH_TO_DATA","PATH_TO_LABEL")
    ```
    以上方法適用於所有圖片都放在同一個資料夾，透過一個.csv檔來區分每筆資料的class。而如果是在跟目錄下分成好幾個資料夾，每個資料夾是一個class的話。例如
    ```
   root
   |____dog
         |____1.jpg
         |____2.jpg
         |____3.jpg
   |____cat
         |____1.jpg
         |____2.jpg
         |____3.jpg
   ```
   

    可以使用torchvision.datasets.imagefloder來更快的實現dataset。如下
    ```python=
    from torchvision import transforms
    import torchvision.datasets as datasets
    my_transform=transforms.Compose([
        transforms.Resize(256),
        transforms.RandomHorizontalFlip(),
        transforms.CenterCrop(224),
        transforms.ToTensor()
    ])
    data_dir='/root'
    data=datasets.ImageFolder(data_dir,my_transform)
    print(data.classes)
    #result:  ['dog','cat']
    ```
    * torch.utils.data.DataLoader
        torch.utils.data.DataLoader在import以後就可以直接使用了，只要將dataset物件傳給它，設定batch_size、shuffle之類的就好。例如
    ```python=
    from torch.utils.data import DataLoader
    
    my_DataLoader=DataLoader(data,batch_size=10,shuffle=True)
    for i,train_data in enumerate(my_DataLoader):
        data,label=train_data
        .
        .
        .
        .
    ```
* train model
    * pytorch在torch.nn裡面有一些現成的loss function，例如L1Loss、MSELoss、CrossEntropyLoss......。
    * torch.optim裡面定義了一些optimizer，optimizer可以基於算出來的梯度對model的參數做更新，並且可以對optimizer設定learning rate、regularization等參數。
    接下來就可以利用loss function和optimizer來對model做training，值得注意的一點是，如果model有使用dropout或是batch normalization，model在train和test之前分別要設定為model.train()或是model.eval()，因為dropout和batch norm在training phase和在testing phase的表現會不一樣。
    * 可以用tqdm來顯示進度條
    ```python=
    import torch
    from torch import nn
    
    epoch=10
    model=Net().cuda()
    criterion=nn.L1Loss()
    optimizer=torch.optim.Adam(model.parameters(),lr=1e-3)
    model.train()
    for e in range(epoch):
        for i,train_data in enumerate(my_DataLoader):
            data,label=train_data
            data=data.cuda()
            label=label.cuda()
            optimizer.zero_grad()
            output=model(data)
            loss=criterion(output,label)
            loss.backward()
            optimizer.step()
    ```
    * torch.optim裡有lr_scheduler，可以隨epoch調整learning rate。例如
    ```python=
    from torch.optim import lr_scheduler
    .
    .
    .
    optimizer=torch.optim.Adam(model.parameters(),lr=1e-3)
    my_lr_shceduler=lr_scheduler.StepLR(optimizer,step_size=5,gamma=0.1)
    #每5個epoch,learning rate變成0.1倍
    for e in range(epoch):
        my_lr_schedular.step()
        .
        .
        .
    ```
* test model
    train完以後做prediction或testing要記得將這部分的code用`with torch.no_grad():`或是`with torch.set_grad_enabled(False):`包起來，還有因為在train的時候是以batch為單位做training，因此資料會是四維的(第幾張圖,RGB,X軸,Y軸)，但如果是prediction的狀態，只有單張圖片的話，需要幫那張圖片多加一個維度，讓它變成是只有一張圖的batch。因此可以利用`tensor=tensor.unsqueeze(0)`讓它在最前面多加一維。例如
    ```python=
    img=cv2.imread(".......")
    img=torch.tensor(img)
    img=img.unsqueeze(0).cuda()
    model.eval()
    with torch.no_grad():
        prediction=model(img)
    #or
    with torch.set_grad_enabled(False):
        prediction=model(img)
    ```
* save/load model:
https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-loading-a-general-checkpoint-for-inference-and-or-resuming-training

# TF2 tutorial
## 基本數值初始化
* 變數的initialization跟pytorch幾乎一樣
```python=
a = tf.ones(shape=(2,3), dtype=tf.int32)

#constant
b = tf.constant([[1, 2, 3], [4, 5, 6]])
print("b =",b);

npvar = np.array(["hello", "world"])
c = tf.constant(npvar)
print("\nc =", c)

d = tf.constant(10.0, shape=[2,5])
print("\nd =", d)

#變數是用Variable

#可以取名字
w = tf.Variable(20., name="my_var01")

initializer = tf.initializers.GlorotUniform()
x = tf.Variable(initializer(shape=(2, 5)), name="my_var02")

y = tf.Variable(tf.zeros([5]), name='my_var03')
```
* 賦值
```python=
w.assign(v)
w.assign_add(v)
```
* 轉成numpy
```python=
w.numpy()
```
## model implementation
### keras sequential API
* keras是TF上層的api，提供一些簡易產生model的方式
```python=
import tensorflow as tf
from tensorflow.keras import layers
model = tf.keras.Sequential()
model.add(layers.Dense(32, activation=‘relu’, input_shape=(784,)) 
model.add(layers.Dense(128, activation='relu')) 
model.add(layers.Dense(10, activation='softmax'))

# or

model = tf.keras.Sequential([
    layers.Dense(32, activation='relu', input_shape=(784,), layers.Dense(128, activation='relu'),
    layers.Dense(10, activation='softmax')
])
```

### Keras functional API
使用functional api可以定義較複雜的模型例如DAG型態的模型
```python=
import tensorflow as tf
from tensorflow.keras import layers
inputs = tf.keras.Input(shape=(784,))
x = layers.Dense(64, activation='relu')(inputs)
x = layers.Dense(64, activation='relu')(x)
outputs = layers.Dense(10, activation='softmax')(x)
model = tf.keras.Model(inputs=inputs, outputs=outputs)
```
使用`model.summary()`可以看mdoel長怎樣
還可以把model的DAG畫出來
```python=
keras.utils.plot_model(model, 'plot.png', show_shapes=True)
```
### model sub-classing
把model包成一個class
```python=
class MyModel(tf.keras.Model):
    def __init__(self, num_classes=10):
        super(MyModel, self).__init__()
        self.dense_1 = layers.Dense(32,activation='relu')
        self.dense_2 = layers.Dense(num_classes,activation='softmax') # Define your forward pass here
    def call(self, inputs):
        x = self.dense_1(inputs) 
        return self.dense_2(x)
    
#也可以把activation放外面
class MyModel(tf.keras.Model):
    def __init__(self, num_classes=10):
        super(MyModel, self).__init__()
        self.dense_1 = layers.Dense(32)
        self.dense_2 = layers.Dense(num_classes,activation='softmax') # Define your forward pass here
    def call(self, inputs):
        x = self.dense_1(inputs) 
        x = tf.nn.relu(x)
        return self.dense_2(x)
```

## training
### Keras built-in method
```python=
model = MyModel()

'''
.compile() is about configuring the training process, 
such as specifying the optimizer, loss, and metrics.
'''
model.compile(optimizer=Adam(), loss=BinaryCrossentropy(), metrics=[AUC(), Precision(), Recall()])
history = model.fit(data, epochs=10, batch_size=128,
                    validation_data=val_data, 
                    callbacks=[EarlyStopping(), TensorBoard(), 
                               ModelCheckpoint()])
results = model.evaluate(test_data, batch_size=128)
```
如果在.compile()裡面加 run_eagerly=True，model就會跑在dynamic graph mode裡。
### GradientTape
自定義training loop
有GradientTape包住的地方TF就會紀錄有關數值的操作，最後就會根據tape記錄到的東西來計算gradient。
```python=
x = tf.constant(3.0)
with tf.GradientTape() as g:
    g.watch(x) 
    y=x*x
    
dy_dx = g.gradient(y, x) # Will compute to 6.0
```

二階微分：
```python=
x = tf.constant(3.0)
with tf.GradientTape() as g:
    g.watch(x)
    with tf.GradientTape() as gg:
        gg.watch(x)
        y=x*x
    dy_dx = gg.gradient(y, x) # Will compute to 6.0 
    
d2y_dx2 = g.gradient(dy_dx, x) # Will compute to 2.0
```

Model training:
```python=
model = MyModel()
loss_fn = tf.keras.losses.SparseCategoricalCrossentropy()
optimizer = tf.keras.optimizers.SGD(learning_rate=0.01)

@tf.function # optional
def train_step(features, labels):
    with tf.GradientTape() as tape: 
        logits = model(features)
        loss = loss_fn(labels, logits)
    grads = tape.gradient(loss, model.trainable_variables)
    optimizer.apply_gradients(zip(grads, model.trainable_variables))
    
for features, labels in data:
    loss = train_step(features, labels)
```

measure loss和accuracy：TF不用像pytorch一樣自己算accuracy之類的，它有提供現成function
```python=
model = ...
optimizer = ...
loss_fn = ...

train_loss = tf.keras.metrics.Mean(name='train_loss')
train_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(name=‘train_accuracy')
                                                            
@tf.function
def train_step(features, labels):
    with tf.GradientTape() as tape: 
        logits = model(features)
        loss = loss_fn(labels, logits)
    grads = tape.gradient(loss, model.trainable_variables)
    optimizer.apply_gradients(zip(grads, model.trainable_variables))
    
    train_loss(loss)
    train_accuracy(labels, logits)
```

```python=
EPOCHS = 5
for epoch in range(EPOCHS):
    # Reset the metrics at the start of the next epoch
    train_loss.reset_states()
    train_accuracy.reset_states()
    
    for images, labels in train_ds: 
        train_step(images, labels)
        
    template = 'Epoch {}, Loss: {}, Accuracy: {}'
    print(template.format(epoch+1,
                            train_loss.result(),
                            train_accuracy.result()*100)
```
總歸一句，tf.keras.metrics的用法：
```python=
m = SomeMetric(...) 
for input in ...:
    m.update_state(input)
    
print('Final result: ', m.result())
```
或
```python=
m = SomeMetric(...) 
for input in ...:
    print('Current result': m(input))
    
print('Final result: ', m.result().numpy())
```
## data
就我的理解，TF這邊的dataset就相當於pytorch包完dataloader的效果。
```python=
AUTOTUNE = tf.data.experimental.AUTOTUNE
X = Y = Z = AUTOTUNE
dataset = tf.data.Dataset.list_files('/path/to/ds/*.jpg')
dataset = dataset.shuffle(NUM_TOTAL_IMAGES)
dataset = dataset.map(read_image_and_label, num_parallel_calls=Y)
dataset = dataset.map(process_image, num_parallel_calls=Z)
dataset = dataset.batch(batch_size=64)
dataset = dataset.prefetch(buffer_size=X) # Enable pipelining
dataset = dataset.cache()#直接load到memory
def read_image_and_label(filepath):
    # label, filepath: 0_28382.jpg
    parts = tf.strings.split(filepath, "_")
    label = tf.strings.to_number(parts[0], out_type=tf.int32) # image
    raw_bytes = tf.io.read_file(filepath)
    return raw_bytes, label


def process_image(raw_bytes, label):
    img = tf.image.decode_jpeg(raw_bytes, channels=3)
    img = tf.image.convert_image_dtype(img, tf.float32) 
    img = tf.image.resize(img, [32, 32])
    img = img / 255.0
    img += tf.random.normal(img.shape, stdev=0.1)
    return img, label
```
![](https://i.imgur.com/f6BB3MU.png)

## save/load
### 自動存
用以下方式就可以把weight和optimizer的參數存下來
```python=
model.save_weights('path_to_my_tf_checkpoint')
model.save_weights('path_to_my_tf_checkpoint.anything') 
model.save_weights('path_to_my_tf_checkpoint', save_format='tf')
```
load的話如果沒有先把model compile過，就只會load model的參數，不會load optimizer的參數
**注意：如果要load optimizer的參數的話，compile的argument要和之前一樣**
```python=
restored_model = create_model()
restored_model.load_weights('path_to_weights')
```
如果是用model sub-classing的方法定義model的話，load weight的方式會和別的不太一樣，因為model sub-classing是lazy initialization，因此input的dimension是餵進第一筆data以後才會知道。所以load weight之前要先餵東西進去。

### callback functions
可以設定一些callback function例如save checkpoint, LR scheduler之類的東西
```python=
cp_callback = tf.keras.callbacks.ModelCheckpoint( 
                filepath=checkpoint_path,
                verbose=1,
                save_weights_only=True,
                period=5)

model.fit(dataset, epochs=10,
        validation_data=validation_dataset, 
        callbacks=[cp_callback]) # Pass callback to training
```
### 手動存
```python=
model = create_model()
opt = tf.keras.optimizers.Adam(0.1)
ckpt = tf.train.Checkpoint(step=tf.Variable(1), optimizer=opt, model=model) 
manager = tf.train.CheckpointManager(ckpt, './tf_ckpts', max_to_keep=3)
save_path = manager.save()
```

## @tf.function
如果在function加這個decorator的話，tf會自動將function轉成graph mode，這樣下次在執行的時候就會快很多。但是變成graph mode會變得很難debug，所以一開始在開發的時後可以先不要放@tf.function，或是在最前面先加上`tf.config.run_functions_eagerly(True)`
![](https://i.imgur.com/FUFQ8fE.png)

## multi GPU training
![](https://i.imgur.com/Ee192dh.png)
## detail of TF
### side effect
在tf.function中，一般的print只有第一次tf在trace（整個function跑過一遍來建立計算圖）的時候會被call到，但一般的print不會被加入計算圖，所以之後再執行到這個function時就不會再被call到了。如果想要一直被call的的話就要換成tf.print。
![](https://i.imgur.com/v7XXozj.png)
就目前的了解，如果某個function是tf.function，但是在裡面被操作的參數都是python的變數而不是tf的變數，那tf就會一直重新產生computation graph，反而造成額外的overhead。
文章有提到盡量不要在tf.function裡面使用list
參考資料：https://jonathan-hui.medium.com/tensorflow-eager-execution-v-s-graph-tf-function-6edaa870b1f1
