# PyTorch cheat sheet
> [name=陳建成] [time=Wed, Mar 25, 2020 7:57 PM][color=#63f2a1]
此處整理 PyTorch 常用的 modules 和 functions 方便快速查詢。
完整且詳細的 docs 請見 [PyTorch 官方文檔（ver 1.2.0）](https://pytorch.org/docs/1.2.0/)。
另外，這裡有兩個版本的 PyTorch 教學 Colaboratory Notebooks，[一個](https://colab.research.google.com/drive/14xSEVRGOPLYNVGfXTnc-vNc1yTp05aDf)和上課教學互相對應，[另一個](https://colab.research.google.com/drive/1PXpKHuETM-xgTatmHpSeZysXkZaXXKja)有更詳細的解說（包含前後處理、視覺化、常用工具等），提供各位參考。
> 對了，拜託各位先 copy 一份到自己雲端硬碟，或是用 Playground 模式，不要干擾到原本的版本，多謝各位合作 :smiley:
[TOC]
:::spoiler 在我們開始前⋯⋯（表示方法說明）
> [color=#ff0000]
> 1. 如果遇到中間出現 `as` 語法的，表示常使用 `import ... as ...` 語法 import，例如：
> ```python
> torch
> ├── nn as nn
> ```
> 這邊表示常常在 coding 時以
> ```python
> import torch.nn as nn
> ```
> 語法 import，其下的 modules、classes 和 functions 也以 `nn.SOMETHING` 形式出現，如同
> ```python
> import numpy as np # np.array...
> import pandas as pd # pd.read_csv...
> ```
>
> 2. 如果今天看到的是有加 `()` 括號的，代表是一個 object 的 method。例如：
> ```python
> torch
> └── (Tensor)
> ├── view
> └── item
> ```
> 這代表一個 `torch.Tensor` object 的 `view` 跟 `item` method
>
> 3. 如果看到有加 `<>` 角括號，代表與前面出現過的是同一個 module，後面會有線條連接到在這個段落中第一次出現的地方。例如：
> ```python
> │ ├── functional as F ────────────────────────────┐
> │ │ ├── relu │
> │ │ ...(Some lines)... │
> │ └──<functional as F> <───────────────────────────┘
> │ └── nll_loss
> ```
> 這邊兩個 `functional` 指的是同一個，不過因為包含兩種不同用途而分開寫
:::
## Tensor Operations [[Docs]](https://pytorch.org/docs/1.2.0/tensors.html)
```python
torch
├── (Tensor)
│ ├── view(*shape) # e.g. x.view(-1, 3, 12)
│ │ ## -1 automatically filled
│ └── item() # get if Tensor is a scalar
│
├── empty(*size) # e.g. x = torch.empty(2, 3)
├── stack(tensors, dim=0)
└── cat(tensors, dim=0)
```
## Data Preparation [[Docs]](https://pytorch.org/docs/1.2.0/data.html)
```python
torch
└── utils
└── data
├── Dataset # A class to override
│ ## `__len__` & `__getitem__`
├── TensorDataset(data_tensor, target_tensor)
├── DataLoader(dataset, batch_size=1,
│ shuffle=False,
│ collate_fn=\
│ <function default_collate>)
│ # define `collate_fn` yourself
└── sampler
├── SequentialSampler(data_source)
└── RandomSampler(data_source)
```
## NN (Neural Network) Model Construction [[Docs]](https://pytorch.org/docs/1.2.0/nn.html)
這是 PyTorch 最主要的 module，docs 比較複雜，分成
- [`torch.nn`](https://pytorch.org/docs/1.2.0/nn.html)
- [`torch.nn.functional`](https://pytorch.org/docs/1.2.0/nn.functional.html)
- [`torch.nn.init`](https://pytorch.org/docs/1.2.0/nn.init.html)
- [`torch.optim`](https://pytorch.org/docs/1.2.0/optim.html)
- [`torch.autograd`](https://pytorch.org/docs/1.2.0/autograd.html)
### Training
```python
torch
├── (Tensor)
│ ├── backward()
│ │
│ ├── cpu()
│ ├── cuda()
│ └── to(torch.device) # x = x.to(device)
├── cuda
│ └── is_available()
│ # if torch.cuda.is_available():
│ ## device = "cuda"
│ ## else: device = "cpu"
│
├── nn as nn
│ │### Models ###
│ ├── Module
│ │ ├── load_state_dict(torch.load(PATH))
│ │ ├── train()
│ │ └── eval()
│ ├── Sequential(layers)
│ │
│ │### Initializations ###
│ ├── init
│ │ └── uniform_(w) # In-place,
│ │ ## w is a `torch.Tensor`.
│ │
│ │### Layers ###
│ ├── Linear(in_feat, out_feat)
│ ├── Dropout(rate)
│ │
│ │### Activations ###
│ ├── Softmax(dim=None)
│ ├── Sigmoid()
│ ├── ReLU()
│ ├── LeakyReLU(negative_slope=0.01)
│ ├── Tanh()
│ ├── GELU()
│ ├── ReLU6() # Model Compression
│ │ # --> Corresponding functions
│ ├── functional as F ────────────────────────────┐
│ │ ├── softmax(input, dim=None) │
│ │ ├── sigmoid(input) │
│ │ ├── relu(input) │
│ │ ├── leaky_relu(input, │
│ │ │ negative_slope=0.01) │
│ │ ├── tanh(input) │
│ │ ├── gelu(input) │
│ │ └── relu6(input) │
│ │ │
│ │### Losses ### │
│ ├── MSELoss() │
│ ├── CrossEntropyLoss() │
│ ├── BCELoss() │
│ ├── NLLLoss() │
│ │ # --> Corresponding functions │
│ └──<functional as F> <───────────────────────────┘
│ ├── mse_loss(input, target)
│ ├── cross_entropy(input,
│ │ target: torch.LongTensor)
│ ├── binary_cross_entropy(input, target)
│ ├── log_softmax(input)
│ └── nll_loss(log_softmax_output, target)
│ # F.nll_loss(F.log_softmax(input), target)
│
│ ### Optimizers ###
├── optim
│ ├── (Optimizer)
│ │ ├── zero_grad()
│ │ ├── step()
│ │ └── state_dict()
│ │
│ ├── SGD(model.parameters(), lr=0.1, momentum=0.9)
│ ├── Adagrad(model.parameters(), lr=0.01,
│ │ lr_decay=0, weight_decay=0,
│ │ initial_accumulator_value=0,eps=1e-10)
│ ├── RMSProp(model.parameters(), lr=0.01,
│ │ alpha=0.99, eps=1e-08, weight_decay=0,
│ │ momentum=0)
│ ├── Adam(model.parameters(), lr=0.001,
│ │ betas=(0.9, 0.999), eps=1e-08,
│ │ weight_decay=0)
│ │
│ └── lr_scheduler
│ └── ReduceLROnPlateau(optimizer)
│
│── load(PATH)
│── save(model, PATH)
│
└── autograd
└── backward(tensors)
```
### Testing
```python
torch
├── nn
│ └── Module
│ ├── load_state_dict(torch.load(PATH))
│ └── eval()
├── optim
│ └── (Optimizer)
│ └── state_dict()
└── no_grad() # with torch.no_grad(): ...
```
## CNN (Convolutional Neural Networks)
- [Convolutional Layers](https://pytorch.org/docs/1.2.0/nn.html#conv2d)
- [Pooling Layers](https://pytorch.org/docs/1.2.0/nn.html#maxpool2d)
- [`torchvision` docs](https://pytorch.org/docs/stable/torchvision/index.html)
```python
torch
├── (Tensor)
│ └── view(*shape)
├── nn
│ │### Layers ###
│ ├── Conv2d(in_channels, out_channels,
│ │ kernel_size, stride=1, padding=0)
│ ├── ConvTranspose2d(in_channels, out_channels,
│ │ kernel_size, stride=1, padding=0,
│ │ output_padding=0)
│ ├── MaxPool2d(kernel_size, stride=None,
│ │ padding=0, dilation=1)
│ │ # stride default: kernel_size
│ ├── BatchNorm2d(num_feat)
│ └── BatchNorm1d(num_feat)
├── stack(tensors, dim=0)
└── cat(tensors, dim=0)
torchvision
├── models as models # Useful pretrained
├── transforms as transforms
│ ├── Compose(transforms) # Wrapper
│ ├── ToPILImage(mode=None)
│ ├── RandomHorizontalFlip(p=0.5)
│ ├── RandomRotation(degrees)
│ ├── ToTensor()
│ └── Resize(size)
└── utils
├── make_grid(tensor, nrow=8, padding=2)
└── save_image(tensor, filename, nrow=8,padding=2)
```
## RNN (Recurrent Neural Networks)
- [Recurrent Layers](https://pytorch.org/docs/1.2.0/nn.html#recurrent-layers)
- [Gensim Word2Vec Docs](https://radimrehurek.com/gensim/models/word2vec.html)
```python
torch
├── nn
│ ├── Embedding(num_embed, embed_dim)
│ │ # embedding = nn.Embedding(
│ │ ## *(w2vmodel.wv.vectors.shape))
│ ├── Parameter(params: torch.FloatTensor)
│ │ # embedding.weight = nn.Parameter(
│ │ ## torch.FloatTensor(w2vmodel.wv.vectors))
│ ├── LongTensor # Feeding Indices of words
│ │
│ ├── LSTM(inp_size, hid_size, num_layers)
│ │ # input: input, (h_0, c_0)
│ └── GRU(inp_size, hid_size, num_layers)
├── stack(tensors, dim=0)
└── cat(tensors, dim=0)
gensim
└── models
└── word2Vec
└── Word2Vec(sentences) # list or words/tokens
```
# Change Log
> 全部的架構太大，不方便查詢，故先隱藏起來 [name=陳建成] [time=Wed, Mar 25, 2020 7:57 PM][color=#63f2a1]
> :::spoiler PyTorch 套件常用部分完整架構
> ```python
> torch
> ├── (Tensor)
> │ ├── view
> │ ├── item
> │ ├── cpu()
> │ ├── cuda()
> │ ├── to(torch.device)
> │ └── backward
> ├── nn
> │ ├── Module
> │ │ ├── load_state_dict
> │ │ ├── train
> │ │ └── eval
> │ ├── Sequential
> │ │ # Layers
> │ ├── Linear
> │ ├── Dropout
> │ │ ## CNN
> │ ├── Conv2d
> │ ├── ConvTranspose2d
> │ ├── MaxPool2d
> │ ├── BatchNorm2d
> │ ├── BatchNorm1d # GAN
> │ │ ## RNN
> │ ├── Embedding
> │ ├── LSTM
> │ ├── GRU
> │ │ # Loss functions
> │ ├── MSELoss
> │ ├── CrossEntropyLoss
> │ ├── BCELoss
> │ │ # Activations
> │ ├── Sigmoid
> │ ├── ReLU
> │ ├── Tanh
> │ ├── ReLU6 # Network Compression
> │ │ # Initializations
> │ ├── init
> │ │ └── uniform_
> │ │
> │ ├── functional as F
> │ │ ├── relu
> │ │ ├── leakyrelu
> │ │ ├── gelu
> │ │ └── nll_loss
> │ │
> │ └── Parameter
> ├── optim
> │ ├── SGD
> │ ├── RMSProp
> │ ├── Adagrad
> │ ├── Adam
> │ ├── AdamW
> │ ├── lr_scheduler
> │ └── (Optimizer)
> │ ├── zero_grad
> │ ├── state_dict
> │ └── step
> ├── utils
> │ └── data
> │ ├── Dataset
> │ ├── TensorDataset
> │ ├── DataLoader
> │ ├── sampler
> │ ├── SequentialSampler
> │ └── RandomSampler
> ├── cuda
> │ └── is_available
> ├── autograd
> │ └── backward
> │ # tensor operation
> ├── no_grad
> ├── empty
> ├── stack
> ├── cat
> │ # model save
> ├── load
> └── save
>
> torchvision
> ├── transforms
> │ ├── Compose
> │ ├── ToPILImage
> │ ├── RandomHorizontalFlip
> │ ├── RandomRotation
> │ ├── ToTensor
> │ └── Resize
> ├── models
> └── utils
> ├── make_grid
> └── save_image
> ```
> :::
Sign in with Wallet
Connect another wallet