---
# System prepended metadata

title: lecture8_note
tags: [Lecture8]

---

# lecture8_note
###### tags: `Lecture8`
## Deep learning hardware
### CPU: 
Fewer cores,but each core is much faster and much more capable; great at sequential tasks

### GPU: 
More cores,but each core is much slower and “dumber”; great for parallel tasks

### TPU: 
Specialized hardware for deep learning

![](https://i.imgur.com/at84msH.png)

![](https://i.imgur.com/9YIFcBI.png)


**reference**:https://zh.wikipedia.org/wiki/%E6%AF%8F%E7%A7%92%E6%B5%AE%E9%BB%9E%E9%81%8B%E7%AE%97%E6%AC%A1%E6%95%B8

### Example in GPU and CPU 
![](https://i.imgur.com/jWcB5Du.png)
GPU在算這種矩陣乘法，他會爆炸性的（blast out and parallel）算出所有元素，把他分割成很多core，再分別去算，效率一定高
但是對於CPU來說，因他是一個一個去算(sequential)，所以會比GPU慢上許多

## Deep learning software
先上一個例子
![](https://i.imgur.com/g8bEr4s.png)
你用numpy寫完
是不是還是得要自己寫backward pass
那是一件很崩潰的事情
功課也寫過
然後就是numpy不能在GPU上面跑
numpy is definitely CPU only

Question:Are there something "framework" can run on GPU,and let you write code in forward pass that looks similar to numpy,and lets you run automatically compute gradients?
這就是大部分的framework的目的
以下介紹第一個
### Pytorch

啊你要用就要裝
可以到[官網](https://pytorch.org/)上面裝一下
![](https://i.imgur.com/yDA9mIM.png)

他在算forward pass的時候跟numpy幾乎是一樣的

![](https://i.imgur.com/5qR6U6z.png)

而且你只要設一下參數
他就可以把你需要的所有gradient回傳！
神奇寶傑！
啊先講一下pytorch裡面有的一些功能屬性
![](https://i.imgur.com/sU8vyJq.png)
### tensor(參code)
他其實就是numpy裡的array
### Autograd(參code)
1.Creating Tensors with requires_grad=True enables autograd
2.Operations on Tensors with requires_grad=True cause PyTorch to build a computational graph
```
import torch

device=torch.device("cpu")

N,D_in,H,D_out=64,1000,100,10
x=torch.randn(N,D_in,device=device)
y=torch.randn(N,D_out,device=device)
#=============================================
w1=torch.randn(D_in,H,device=device,requires_grad=True)
#Creating Tensors with requires_grad=True enables autograd
w2=torch.randn(H,D_out,device=device,requires_grad=True)
#=============================================
learning_rate=1e-6

for t in range(500):
#     Forward pass looks exactly
# the same as before, but we
# don’t need to track
# intermediate values -
# PyTorch keeps track of
# them for us in the graph
    h=x.mm(w1)
    h_relu=h.clamp(min=0)
    y_pred=h_relu.mm(w2)
    loss=(y_pred-y).pow(2).sum()
#=============================================
#    原本的backwardpass 很複雜
#     grad_y_pred=2.0*(y_pred-y)
#     grad_w2=h_relu.t().mm(grad_y_pred)
#     grad_h_relu=grad_y_pred.mm(w2.t())
#     grad_h=grad_h_relu.clone()
#     grad_h[h<0]=0
#     grad_w1=x.t().mm(grad_h)
#=============================================

#但如果你用pytorch會簡單很多
    loss.backward()
#     Compute gradient of loss with respect to w1 and w2
#=============================================
with torch.no_grad():
    
    w1-=learning_rate*w1.grad
    w2-=learning_rate*w2.grad
    w1.grad.zero_()
    w2.grad.zero_()
```
### New autograd（參圖）
平常是用不太到啦
這個意思就是你可以自己去寫一個新的
Define your own autograd functions by writing forward and backward functions for Tensors
Define a helper function to make it easy to use the new function
![](https://i.imgur.com/Y5hc0Vh.jpg)
### nn（參code）
就像tensoflow裡邊的keras一樣
恩恩這個package提供給你更高級的API使用
```
import torch

device=torch.device("cpu")

N,D_in,H,D_out=64,1000,100,10
x=torch.randn(N,D_in,device=device)
y=torch.randn(N,D_out,device=device)
#===============================================
# Define our model as a
# sequence of layers; each
# layer is an object that
# holds learnable weights
#===============================================
model=torch.nn.Sequential(
        torch.nn.Linear(D_in,H),
        torch.nn.ReLU(),
        torch.nn.Linear(H,D_out)
        )
#===============================================
learning_rate=1e-2

for t in range(500):
#===============================================
#     Forward pass: feed data to
# model, and compute loss
    y_pred=model(x)
    loss=torch.nn.functional.mse_loss(y_pred,y)
#     torch.nn.functional has useful
# helpers like loss functions
#reference=====>https://pytorch-cn.readthedocs.io/zh/latest/package_references/functional/#_1
#===============================================
    loss.backward()
#     Backward pass: compute
# gradient with respect to all
# model weights (they have
# requires_grad=True)
#===============================================  
    with torch.no_grad():
        for param in model.parameters():
            param-=learning_rate*param.grad
    model.zero_grad()
#     Make gradient step on
# each model parameter
# (with gradients disabled)

```
### optimizer(參code)
```
import torch

device=torch.device("cpu")

N,D_in,H,D_out=64,1000,100,10
x=torch.randn(N,D_in,device=device)
y=torch.randn(N,D_out,device=device)

model=torch.nn.Sequential(
        torch.nn.Linear(D_in,H),
        torch.nn.ReLU(),
        torch.nn.Linear(H,D_out)
        )

learning_rate=1e-4
#===============================================
# Use an optimizer for
# different update rules
optimizer=torch.optim.Adam(model.parameters(),
                           lr=learing_rate)
#===============================================
for t in range(500):

    y_pred=model(x)
    loss=torch.nn.functional.mse_loss(y_pred,y)

    loss.backward()
#==============================================
#     After computing gradients, use
# optimizer to update params
# and zero gradients
    optimizer.step()
    optimizer.zero_grad()
#==============================================

```
### New module(參code)
You can define your own Modules using autograd!
Modules can contain weights or other modules
```
import torch
#==============================================
#Define our whole model
# as a single Module
class TwoLayerNet(torch.nn.module):
    def __init__(self,D_in,H,D_out):
        super(TwoLayerNet,self).__init()
        self.linear1=torch.nn.Linear(D_in,H)
        self.linear2=torch.nn.Linear(H,D_out)
    def forward(self,x):#Define forward pass using child modules
        h_relu=self.linear1(x).clamp(min=0)
        y_pred=self.linear2(h_relu)
        return y_pred
#No need to define
# backward - autograd will
# handle it
    
#==============================================
N,D_in,H,D_out=64,1000,100,10
x=torch.randn(N,D_in,device=device)
y=torch.randn(N,D_out,device=device)

#==============================================
# Construct and train an
# instance of our model
model=torch.TwoLayerNet(D_in,H,D_out)

optimizer=torch.optim.SGD(model.parameters(),lr=1e-4)

learning_rate=1e-4

for t in range(500):

    y_pred=model(x)
    loss=torch.nn.functional.mse_loss(y_pred,y)

    loss.backward()
    
    optimizer.step()
    optimizer.zero_grad()
#==============================================

```
### Dataloader（參圖）
![](https://i.imgur.com/IOfpSuG.png)

![](https://i.imgur.com/e8N1emL.png)
### pretrain model
![](https://i.imgur.com/8IYKDQc.png)
### Visdom
是在pytorch中一個visualize的工具
reference:
https://github.com/facebookresearch/visdom
https://zhuanlan.zhihu.com/p/32025746

### static dynamic optimization
#### dynamic:
![](https://i.imgur.com/wEYCyxR.png)

#### static:
![](https://i.imgur.com/gmcgLSC.jpg)

ps:This note only focus on pytorch rather than tensoflow.















