owned this note
owned this note
Published
Linked with GitHub
PyTorch 1.4 Tutorial
===
---
## What is PyTorch?
PyTorch is an open-source machine learning library for Python... [[Wiki](https://en.wikipedia.org/wiki/PyTorch)]
- Low-level API
- Automatic Differentiation
- GPU Acceleration
- Dynamic Computation Graph
PyTorch e-book
- https://pytorch.org/deep-learning-with-pytorch
---
## Outline
- Installation
- Basic Data Type
- Data Loading and Processing
- Building Neural Network
- Training and Testing
---
## Installation
- Install `pytorch` and `torchvision` on official site:
https://pytorch.org/get-started/locally/#start-locally
---
## Before Installing...
- Install Python from python.org or Anaconda/Miniconda
- Choose a package manager `pip`/`conda`
- Install CUDA toolkit (if GPU supports are needed)
- NVIDIA Graphics Card
---
## Example
Use `conda` as package manager
- Run `conda create -n pytorch python=3` to create a virtual environment `pytorch`
- Run `source activate pytorch` to activate my virtual environment `pytorch`
- Run `conda install pytorch torchvision cudatoolkit=9.0 -c pytorch` to install
---
## Test Installation
- Run `python -c 'import torch; print(torch.__version__)'` on Bash
```bash
$ python -c 'import torch; print(torch.__version__)'
1.4.0
```
## Test CUDA Installation (Test GPU Support)
- Run `python -c 'import torch; print(torch.cuda.is_available())'`
```bash
$ python -c 'import torch; print(torch.cuda.is_available())'
True
```
---
## Basic Data Type: `torch.Tensor`
- **Tensor**: almost equal to multi-dimensional arrays
- The **shape** of a tensor: the number of dimensions for each rank
- Examples
- the shape of a $255 \times 255$ gray scale image is $(255, 255)$
- the shape of a $255 \times 255$ RGB color image is $(3, 255, 255)$, the number of channels is $3$.
- the shape of a 10-batch $255 \times 255$ RGB color image is $(10, 3, 255, 255)$
---
[`torch.Tensor`](https://pytorch.org/docs/stable/tensors.html)
- Almost equal to `numpy.ndarray` [[NumPy](http://www.numpy.org/)]
- Support construct by built-in `list`
```python
>>> import torch
>>> x = torch.tensor([[5, 4], [8, 7]])
>>> x.shape
torch.Size([2, 2])
```
---
## Gradient Computation
1. Set `requires_grad=True` in a `torch.Tensor` object
2. Context manager (`with...as...` statement)
- `torch.set_grad_enabled(True)` or `torch.set_grad_enabled(False)`
- `torch.enable_grad()`
- `torch.no_grad()`
---
## An Example of Gradient Computation
- $f(x,y)=x^2 + 2y$
```python
>>> def f(x, y):
... return x.pow(2) + 2*y
```
- Set $x=8, y=7$ and `requires_grad=True` if gradient computation is needed
```python
>>> x = torch.tensor([8.], requires_grad=True)
>>> y = torch.tensor([7.], requires_grad=True)
```
- $f(8, 7)=78$
```python
>>> f(x, y)
tensor([78.], grad_fn=<AddBackward0>)
```
$$
\nabla f= \left( \dfrac{\partial f}{\partial x}, \dfrac{\partial f}{\partial y} \right) = \left( 2x, 2 \right)
$$
$$
\Rightarrow \nabla f(8, 7)=(16, 2)
$$
```python
>>> f(x, y).backward()
>>> x
tensor([8.], requires_grad=True)
>>> x.grad
tensor([16.])
>>> y.grad
tensor([2.])
```
---
# GPU Computation
- Default device is CPU
```python
>>> x = torch.tensor([8.0])
>>> x.device
device(type='cpu')
```
- Transfer data to GPU
```
>>> x.to(torch.device('cuda'))
tensor([8.], device='cuda:0')
```
- Specify GPU ID
```
>>> x.to(torch.device('cuda:1'))
tensor([8.], device='cuda:1')
```
---
## Data Loading and Processing
- `torchvision.transforms`
- `torch.utils.data.Dataset`
- `torch.utils.data.Dataloader`
---
## `torchvision.transforms`
Transforms are common image transformations. They can be chained together using `Compose`.
- Transforms on `PIL.Image`
- `ColorJitter(brightness=0, contrast=0, saturation=0, hue=0)`
- `RandomCrop(size, padding=None, pad_if_needed=False, fill=0, padding_mode='constant')`
- `Resize(size, interpolation=2)`
- Transforms on `torch.Tensor`
- `Normalize(mean, std, inplace=False)`
- `RandomErasing(p=0.5, scale=(0.02, 0.33), ratio=(0.3, 3.3), value=0, inplace=False)`
- Conversion Transforms
- `ToPILImage(mode=None)`
- Convert a `torch.Tensor` or an `numpy.ndarray` to `PIL.Image`.
- `ToTensor`
- Convert a `PIL.Image` or `numpy.ndarray` to `torch.Tensor`.
- Functional Transforms
---
## An Example of `torchvision.transforms`
```python=
data_transform = transforms.Compose([
transforms.RandomResizedCrop(224),
transforms.ToTensor(),
transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
])
```
---
## `torch.utils.data.Dataset`
[`torch.utils.data.Dataset`](https://pytorch.org/docs/stable/data.html#torch.utils.data.Dataset)
- An abstract class representing a Dataset.
To use...
1. derive `torch.utils.data.Dataset`
1. implement
- `__len__` for the size of the dataset
- `__getitem__` to access the dataset with index
---
## An Example of `torch.utils.data.Dataset`
[[Reference](https://github.com/remorsecs/Kaggle-Plant-Seedlings-Classification-Example/blob/master/dataset.py)]
- derive `torch.utils.data.Dataset`
```python
from torch.utils.data import Dataset
class PlantSeedlingDataset(Dataset):
def __init__(self, root_dir, transform=None):
self.root_dir = Path(root_dir)
self.x = []
self.y = []
self.transform = transform
self.num_classes = 0
if self.root_dir.name == 'train':
for i, _dir in enumerate(self.root_dir.glob('*')):
for file in _dir.glob('*'):
self.x.append(file)
self.y.append(i)
self.num_classes += 1
...
```
[[Reference](https://github.com/remorsecs/Kaggle-Plant-Seedlings-Classification-Example/blob/master/dataset.py)]
- Implement `__len__` for the size of the dataset
```python=22
def __len__(self):
return len(self.x)
```
- Implement `__getitem__` to access the dataset with index
```python=25
def __getitem__(self, index):
image = Image.open(self.x[index]).convert('RGB')
if self.transform:
image = self.transform(image)
return image, self.y[index]
```
---
## `torch.utils.data.DataLoader`
[`torch.utils.data.DataLoader`](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader)
- An interface for data loader
- Usage
```python
torch.utils.data.DataLoader(
dataset,
batch_size=1,
shuffle=False,
sampler=None,
batch_sampler=None,
num_workers=0,
...
)
```
---
## An Example of `torch.utils.data.DataLoader`
[[Reference](https://github.com/remorsecs/Kaggle-Plant-Seedlings-Classification-Example/blob/master/train.py)]
```python
train_set = PlantSeedlingDataset(
Path(DATASET_ROOT).joinpath('train'),
data_transform,
)
data_loader = DataLoader(
dataset=train_set,
batch_size=32,
shuffle=True,
)
```
---
## Building Neural Network
To build a neural network, you need
- model
- loss function
- optimizer
---
## Building a Model with `torch.nn.Module`
[`torch.nn.Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module)
- Base class for all neural network modules.
To use...
1. derive `torch.nn.Module`
1. implement `forward`
2. call it as function
- don't call `forward()` directly
---
## An Example of `torch.nn.Module`
[[Reference](https://pytorch.org/docs/stable/nn.html#torch.nn.Module)]
```python
import torch.nn as nn
import torch.nn.functional as F
class Model(nn.Module):
def __init__(self):
super(Model, self).__init__()
self.conv1 = nn.Conv2d(1, 20, 5)
self.conv2 = nn.Conv2d(20, 20, 5)
def forward(self, x):
x = F.relu(self.conv1(x))
return F.relu(self.conv2(x))
model = Model()
y = model(x)
```
---
## Loss Function
`torch.nn.*Loss`, e.g.
- `torch.nn.L1Loss`
- `torch.nn.MSELoss`
- ... see: https://pytorch.org/docs/stable/nn.html#loss-functions
`torch.nn.functional.*`, e.g.
- `torch.nn.functional.binary_cross_entropy`
- `torch.nn.functional.binary_cross_entropy_with_logits`
- ... see: https://pytorch.org/docs/stable/nn.html#id51
---
## Optimizer in `torch.optim`
`torch.optim` is a package implementing various optimization algorithms.
- Stochastic Gradient Descent (SGD) example
```python
optimizer = torch.optim.SGD(
model.parameters(),
lr=0.01,
momentum=0.9,
)
```
---
## Training
Loop over the dataset
- Fetch data from `dataloader` (training set)
- Forward
- pass data into model and loss function
- Backward
- compute the gradient of loss value and use optimizer to update parameters
## Testing
Loop over the dataset
- Fetch data from `dataloader` (test set)
- Forward
- pass data into model to get the results
- Evaluate
---
## A Simple Example of Training
```python
def train(model, device, dataloader, optimizer, num_epochs):
for epoch in range(num_epochs):
for (inputs, targets) in dataloader:
inputs = inputs.to(device)
targets = targets.to(device)
# forward
outputs = model(inputs)
loss = F.cross_entropy(outputs, targets)
# backward
optimizer.zero_grad()
loss.backward()
optimizer.step()
```
---
## A Simple Example of Testing
```python
def test(model, device, dataloader, optimizer):
with torch.no_grad():
for (inputs, targets) in dataloader:
inputs = inputs.to(device)
targets = targets.to(device)
# forward
outputs = model(inputs)
loss = F.cross_entropy(outputs, targets)
# evaluate outputs
```
---
###### tags: `python` `guide`
Google Drive
Gist
Clipboard
Sign in with Wallet
