# PyTorch cheat sheet > [name=陳建成] [time=Wed, Mar 25, 2020 7:57 PM][color=#63f2a1] 此處整理 PyTorch 常用的 modules 和 functions 方便快速查詢。 完整且詳細的 docs 請見 [PyTorch 官方文檔(ver 1.2.0)](https://pytorch.org/docs/1.2.0/)。 另外,這裡有兩個版本的 PyTorch 教學 Colaboratory Notebooks,[一個](https://colab.research.google.com/drive/14xSEVRGOPLYNVGfXTnc-vNc1yTp05aDf)和上課教學互相對應,[另一個](https://colab.research.google.com/drive/1PXpKHuETM-xgTatmHpSeZysXkZaXXKja)有更詳細的解說(包含前後處理、視覺化、常用工具等),提供各位參考。 > 對了,拜託各位先 copy 一份到自己雲端硬碟,或是用 Playground 模式,不要干擾到原本的版本,多謝各位合作 :smiley: [TOC] :::spoiler 在我們開始前⋯⋯(表示方法說明) > [color=#ff0000] > 1. 如果遇到中間出現 `as` 語法的,表示常使用 `import ... as ...` 語法 import,例如: > ```python > torch > ├── nn as nn > ``` > 這邊表示常常在 coding 時以 > ```python > import torch.nn as nn > ``` > 語法 import,其下的 modules、classes 和 functions 也以 `nn.SOMETHING` 形式出現,如同 > ```python > import numpy as np # np.array... > import pandas as pd # pd.read_csv... > ``` > > 2. 如果今天看到的是有加 `()` 括號的,代表是一個 object 的 method。例如: > ```python > torch > └── (Tensor) > ├── view > └── item > ``` > 這代表一個 `torch.Tensor` object 的 `view` 跟 `item` method > > 3. 如果看到有加 `<>` 角括號,代表與前面出現過的是同一個 module,後面會有線條連接到在這個段落中第一次出現的地方。例如: > ```python > │ ├── functional as F ────────────────────────────┐ > │ │ ├── relu │ > │ │ ...(Some lines)... │ > │ └──<functional as F> <───────────────────────────┘ > │ └── nll_loss > ``` > 這邊兩個 `functional` 指的是同一個,不過因為包含兩種不同用途而分開寫 ::: ## Tensor Operations [[Docs]](https://pytorch.org/docs/1.2.0/tensors.html) ```python torch ├── (Tensor) │ ├── view(*shape) # e.g. x.view(-1, 3, 12) │ │ ## -1 automatically filled │ └── item() # get if Tensor is a scalar │ ├── empty(*size) # e.g. x = torch.empty(2, 3) ├── stack(tensors, dim=0) └── cat(tensors, dim=0) ``` ## Data Preparation [[Docs]](https://pytorch.org/docs/1.2.0/data.html) ```python torch └── utils └── data ├── Dataset # A class to override │ ## `__len__` & `__getitem__` ├── TensorDataset(data_tensor, target_tensor) ├── DataLoader(dataset, batch_size=1, │ shuffle=False, │ collate_fn=\ │ <function default_collate>) │ # define `collate_fn` yourself └── sampler ├── SequentialSampler(data_source) └── RandomSampler(data_source) ``` ## NN (Neural Network) Model Construction [[Docs]](https://pytorch.org/docs/1.2.0/nn.html) 這是 PyTorch 最主要的 module,docs 比較複雜,分成 - [`torch.nn`](https://pytorch.org/docs/1.2.0/nn.html) - [`torch.nn.functional`](https://pytorch.org/docs/1.2.0/nn.functional.html) - [`torch.nn.init`](https://pytorch.org/docs/1.2.0/nn.init.html) - [`torch.optim`](https://pytorch.org/docs/1.2.0/optim.html) - [`torch.autograd`](https://pytorch.org/docs/1.2.0/autograd.html) ### Training ```python torch ├── (Tensor) │ ├── backward() │ │ │ ├── cpu() │ ├── cuda() │ └── to(torch.device) # x = x.to(device) ├── cuda │ └── is_available() │ # if torch.cuda.is_available(): │ ## device = "cuda" │ ## else: device = "cpu" │ ├── nn as nn │ │### Models ### │ ├── Module │ │ ├── load_state_dict(torch.load(PATH)) │ │ ├── train() │ │ └── eval() │ ├── Sequential(layers) │ │ │ │### Initializations ### │ ├── init │ │ └── uniform_(w) # In-place, │ │ ## w is a `torch.Tensor`. │ │ │ │### Layers ### │ ├── Linear(in_feat, out_feat) │ ├── Dropout(rate) │ │ │ │### Activations ### │ ├── Softmax(dim=None) │ ├── Sigmoid() │ ├── ReLU() │ ├── LeakyReLU(negative_slope=0.01) │ ├── Tanh() │ ├── GELU() │ ├── ReLU6() # Model Compression │ │ # --> Corresponding functions │ ├── functional as F ────────────────────────────┐ │ │ ├── softmax(input, dim=None) │ │ │ ├── sigmoid(input) │ │ │ ├── relu(input) │ │ │ ├── leaky_relu(input, │ │ │ │ negative_slope=0.01) │ │ │ ├── tanh(input) │ │ │ ├── gelu(input) │ │ │ └── relu6(input) │ │ │ │ │ │### Losses ### │ │ ├── MSELoss() │ │ ├── CrossEntropyLoss() │ │ ├── BCELoss() │ │ ├── NLLLoss() │ │ │ # --> Corresponding functions │ │ └──<functional as F> <───────────────────────────┘ │ ├── mse_loss(input, target) │ ├── cross_entropy(input, │ │ target: torch.LongTensor) │ ├── binary_cross_entropy(input, target) │ ├── log_softmax(input) │ └── nll_loss(log_softmax_output, target) │ # F.nll_loss(F.log_softmax(input), target) │ │ ### Optimizers ### ├── optim │ ├── (Optimizer) │ │ ├── zero_grad() │ │ ├── step() │ │ └── state_dict() │ │ │ ├── SGD(model.parameters(), lr=0.1, momentum=0.9) │ ├── Adagrad(model.parameters(), lr=0.01, │ │ lr_decay=0, weight_decay=0, │ │ initial_accumulator_value=0,eps=1e-10) │ ├── RMSProp(model.parameters(), lr=0.01, │ │ alpha=0.99, eps=1e-08, weight_decay=0, │ │ momentum=0) │ ├── Adam(model.parameters(), lr=0.001, │ │ betas=(0.9, 0.999), eps=1e-08, │ │ weight_decay=0) │ │ │ └── lr_scheduler │ └── ReduceLROnPlateau(optimizer) │ │── load(PATH) │── save(model, PATH) │ └── autograd └── backward(tensors) ``` ### Testing ```python torch ├── nn │ └── Module │ ├── load_state_dict(torch.load(PATH)) │ └── eval() ├── optim │ └── (Optimizer) │ └── state_dict() └── no_grad() # with torch.no_grad(): ... ``` ## CNN (Convolutional Neural Networks) - [Convolutional Layers](https://pytorch.org/docs/1.2.0/nn.html#conv2d) - [Pooling Layers](https://pytorch.org/docs/1.2.0/nn.html#maxpool2d) - [`torchvision` docs](https://pytorch.org/docs/stable/torchvision/index.html) ```python torch ├── (Tensor) │ └── view(*shape) ├── nn │ │### Layers ### │ ├── Conv2d(in_channels, out_channels, │ │ kernel_size, stride=1, padding=0) │ ├── ConvTranspose2d(in_channels, out_channels, │ │ kernel_size, stride=1, padding=0, │ │ output_padding=0) │ ├── MaxPool2d(kernel_size, stride=None, │ │ padding=0, dilation=1) │ │ # stride default: kernel_size │ ├── BatchNorm2d(num_feat) │ └── BatchNorm1d(num_feat) ├── stack(tensors, dim=0) └── cat(tensors, dim=0) torchvision ├── models as models # Useful pretrained ├── transforms as transforms │ ├── Compose(transforms) # Wrapper │ ├── ToPILImage(mode=None) │ ├── RandomHorizontalFlip(p=0.5) │ ├── RandomRotation(degrees) │ ├── ToTensor() │ └── Resize(size) └── utils ├── make_grid(tensor, nrow=8, padding=2) └── save_image(tensor, filename, nrow=8,padding=2) ``` ## RNN (Recurrent Neural Networks) - [Recurrent Layers](https://pytorch.org/docs/1.2.0/nn.html#recurrent-layers) - [Gensim Word2Vec Docs](https://radimrehurek.com/gensim/models/word2vec.html) ```python torch ├── nn │ ├── Embedding(num_embed, embed_dim) │ │ # embedding = nn.Embedding( │ │ ## *(w2vmodel.wv.vectors.shape)) │ ├── Parameter(params: torch.FloatTensor) │ │ # embedding.weight = nn.Parameter( │ │ ## torch.FloatTensor(w2vmodel.wv.vectors)) │ ├── LongTensor # Feeding Indices of words │ │ │ ├── LSTM(inp_size, hid_size, num_layers) │ │ # input: input, (h_0, c_0) │ └── GRU(inp_size, hid_size, num_layers) ├── stack(tensors, dim=0) └── cat(tensors, dim=0) gensim └── models └── word2Vec └── Word2Vec(sentences) # list or words/tokens ``` # Change Log > 全部的架構太大,不方便查詢,故先隱藏起來 [name=陳建成] [time=Wed, Mar 25, 2020 7:57 PM][color=#63f2a1] > :::spoiler PyTorch 套件常用部分完整架構 > ```python > torch > ├── (Tensor) > │ ├── view > │ ├── item > │ ├── cpu() > │ ├── cuda() > │ ├── to(torch.device) > │ └── backward > ├── nn > │ ├── Module > │ │ ├── load_state_dict > │ │ ├── train > │ │ └── eval > │ ├── Sequential > │ │ # Layers > │ ├── Linear > │ ├── Dropout > │ │ ## CNN > │ ├── Conv2d > │ ├── ConvTranspose2d > │ ├── MaxPool2d > │ ├── BatchNorm2d > │ ├── BatchNorm1d # GAN > │ │ ## RNN > │ ├── Embedding > │ ├── LSTM > │ ├── GRU > │ │ # Loss functions > │ ├── MSELoss > │ ├── CrossEntropyLoss > │ ├── BCELoss > │ │ # Activations > │ ├── Sigmoid > │ ├── ReLU > │ ├── Tanh > │ ├── ReLU6 # Network Compression > │ │ # Initializations > │ ├── init > │ │ └── uniform_ > │ │ > │ ├── functional as F > │ │ ├── relu > │ │ ├── leakyrelu > │ │ ├── gelu > │ │ └── nll_loss > │ │ > │ └── Parameter > ├── optim > │ ├── SGD > │ ├── RMSProp > │ ├── Adagrad > │ ├── Adam > │ ├── AdamW > │ ├── lr_scheduler > │ └── (Optimizer) > │ ├── zero_grad > │ ├── state_dict > │ └── step > ├── utils > │ └── data > │ ├── Dataset > │ ├── TensorDataset > │ ├── DataLoader > │ ├── sampler > │ ├── SequentialSampler > │ └── RandomSampler > ├── cuda > │ └── is_available > ├── autograd > │ └── backward > │ # tensor operation > ├── no_grad > ├── empty > ├── stack > ├── cat > │ # model save > ├── load > └── save > > torchvision > ├── transforms > │ ├── Compose > │ ├── ToPILImage > │ ├── RandomHorizontalFlip > │ ├── RandomRotation > │ ├── ToTensor > │ └── Resize > ├── models > └── utils > ├── make_grid > └── save_image > ``` > :::