What is PyTorch?

PyTorch is an open-source machine learning library for Python… [Wiki]

• Low-level API
• Automatic Differentiation
• GPU Acceleration
• Dynamic Computation Graph

PyTorch e-book

• https://pytorch.org/deep-learning-with-pytorch

Outline

• Installation
• Basic Data Type
• Data Loading and Processing
• Building Neural Network
• Training and Testing

Installation

• Install pytorch and torchvision on official site:
  https://pytorch.org/get-started/locally/#start-locally
  

Before Installing…

• Install Python from python.org or Anaconda/Miniconda
• Choose a package manager pip/conda
• Install CUDA toolkit (if GPU supports are needed)
  • NVIDIA Graphics Card

Example

Use conda as package manager

• Run conda create -n pytorch python=3 to create a virtual environment pytorch
• Run source activate pytorch to activate my virtual environment pytorch
• Run conda install pytorch torchvision cudatoolkit=9.0 -c pytorch to install

Test Installation

• Run python -c 'import torch; print(torch.__version__)' on Bash

$ python -c 'import torch; print(torch.__version__)'
1.4.0

Test CUDA Installation (Test GPU Support)

• Run python -c 'import torch; print(torch.cuda.is_available())'

$ python -c 'import torch; print(torch.cuda.is_available())'
True

Basic Data Type: torch.Tensor

• Tensor: almost equal to multi-dimensional arrays
• The shape of a tensor: the number of dimensions for each rank
• Examples
  • the shape of a \(255 \times 255\) gray scale image is \((255, 255)\)
  • the shape of a \(255 \times 255\) RGB color image is \((3, 255, 255)\), the number of channels is \(3\).
  • the shape of a 10-batch \(255 \times 255\) RGB color image is \((10, 3, 255, 255)\)

torch.Tensor

• Almost equal to numpy.ndarray [NumPy]
• Support construct by built-in list

>>> import torch
>>> x = torch.tensor([[5, 4], [8, 7]])
>>> x.shape
torch.Size([2, 2])

Gradient Computation

1. Set requires_grad=True in a torch.Tensor object
2. Context manager (with...as... statement)
   • torch.set_grad_enabled(True) or torch.set_grad_enabled(False)
   • torch.enable_grad()
   • torch.no_grad()

An Example of Gradient Computation

• \(f(x,y)=x^2 + 2y\)

>>> def f(x, y):
...     return x.pow(2) + 2*y

• Set \(x=8, y=7\) and requires_grad=True if gradient computation is needed

>>> x = torch.tensor([8.], requires_grad=True)
>>> y = torch.tensor([7.], requires_grad=True)

• \(f(8, 7)=78\)

>>> f(x, y)
tensor([78.], grad_fn=<AddBackward0>)

\[ \nabla f= \left( \dfrac{\partial f}{\partial x}, \dfrac{\partial f}{\partial y} \right) = \left( 2x, 2 \right) \]
\[ \Rightarrow \nabla f(8, 7)=(16, 2) \]

>>> f(x, y).backward()
>>> x
tensor([8.], requires_grad=True)
>>> x.grad
tensor([16.])
>>> y.grad
tensor([2.])

GPU Computation

• Default device is CPU

>>> x = torch.tensor([8.0])
>>> x.device
device(type='cpu')

• Transfer data to GPU

>>> x.to(torch.device('cuda'))
tensor([8.], device='cuda:0')

• Specify GPU ID

>>> x.to(torch.device('cuda:1'))
tensor([8.], device='cuda:1')

Data Loading and Processing

• torchvision.transforms
• torch.utils.data.Dataset
• torch.utils.data.Dataloader

torchvision.transforms

Transforms are common image transformations. They can be chained together using Compose.

• Transforms on PIL.Image
  • ColorJitter(brightness=0, contrast=0, saturation=0, hue=0)
  • RandomCrop(size, padding=None, pad_if_needed=False, fill=0, padding_mode='constant')
  • Resize(size, interpolation=2)
• Transforms on torch.Tensor
  • Normalize(mean, std, inplace=False)
  • RandomErasing(p=0.5, scale=(0.02, 0.33), ratio=(0.3, 3.3), value=0, inplace=False)
• Conversion Transforms
  • ToPILImage(mode=None)
    • Convert a torch.Tensor or an numpy.ndarray to PIL.Image.
  • ToTensor
    • Convert a PIL.Image or numpy.ndarray to torch.Tensor.
• Functional Transforms

An Example of torchvision.transforms

data_transform = transforms.Compose([
    transforms.RandomResizedCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
])

torch.utils.data.Dataset

torch.utils.data.Dataset

• An abstract class representing a Dataset.

To use…

1. derive torch.utils.data.Dataset
2. implement
   • __len__ for the size of the dataset
   • __getitem__ to access the dataset with index

An Example of torch.utils.data.Dataset

[Reference]

• derive torch.utils.data.Dataset

from torch.utils.data import Dataset

class PlantSeedlingDataset(Dataset):
    def __init__(self, root_dir, transform=None):
        self.root_dir = Path(root_dir)
        self.x = []
        self.y = []
        self.transform = transform
        self.num_classes = 0

        if self.root_dir.name == 'train':
            for i, _dir in enumerate(self.root_dir.glob('*')):
                for file in _dir.glob('*'):
                    self.x.append(file)
                    self.y.append(i)

                self.num_classes += 1
        ...

[Reference]

• Implement __len__ for the size of the dataset

    def __len__(self):
        return len(self.x)

• Implement __getitem__ to access the dataset with index

    def __getitem__(self, index):
        image = Image.open(self.x[index]).convert('RGB')
 
        if self.transform:
            image = self.transform(image)
 
        return image, self.y[index]

torch.utils.data.DataLoader

torch.utils.data.DataLoader

• An interface for data loader
• Usage

torch.utils.data.DataLoader(
    dataset,
    batch_size=1,
    shuffle=False,
    sampler=None,
    batch_sampler=None,
    num_workers=0,
    ...
)

An Example of torch.utils.data.DataLoader

[Reference]

    train_set = PlantSeedlingDataset(
                    Path(DATASET_ROOT).joinpath('train'),
                    data_transform,
                )
    data_loader = DataLoader(
                      dataset=train_set,
                      batch_size=32,
                      shuffle=True,
                  )

Building Neural Network

To build a neural network, you need

• model
• loss function
• optimizer

Building a Model with torch.nn.Module

torch.nn.Module

• Base class for all neural network modules.

To use…

1. derive torch.nn.Module
2. implement forward
3. call it as function
   • don't call forward() directly

An Example of torch.nn.Module

[Reference]

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
       x = F.relu(self.conv1(x))
       return F.relu(self.conv2(x))
       
model = Model()
y = model(x)

Loss Function

torch.nn.*Loss, e.g.

• torch.nn.L1Loss
• torch.nn.MSELoss
• … see: https://pytorch.org/docs/stable/nn.html#loss-functions

torch.nn.functional.*, e.g.

• torch.nn.functional.binary_cross_entropy
• torch.nn.functional.binary_cross_entropy_with_logits
• … see: https://pytorch.org/docs/stable/nn.html#id51

Optimizer in torch.optim

torch.optim is a package implementing various optimization algorithms.

• Stochastic Gradient Descent (SGD) example

optimizer = torch.optim.SGD(
    model.parameters(), 
    lr=0.01,
    momentum=0.9,
)

Training

Loop over the dataset

• Fetch data from dataloader (training set)
• Forward
  • pass data into model and loss function
• Backward
  • compute the gradient of loss value and use optimizer to update parameters

Testing

Loop over the dataset

• Fetch data from dataloader (test set)
• Forward
  • pass data into model to get the results
• Evaluate

A Simple Example of Training

def train(model, device, dataloader, optimizer, num_epochs):
    for epoch in range(num_epochs):
        for (inputs, targets) in dataloader:
            inputs = inputs.to(device)
            targets = targets.to(device)

            # forward
            outputs = model(inputs)
            loss = F.cross_entropy(outputs, targets)

            # backward
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

A Simple Example of Testing

def test(model, device, dataloader, optimizer):
    with torch.no_grad():
        for (inputs, targets) in dataloader:
            inputs = inputs.to(device)
            targets = targets.to(device)

            # forward
            outputs = model(inputs)
            loss = F.cross_entropy(outputs, targets)

            # evaluate outputs

tags: python guide