PyTorch 1.4 Tutorial === --- ## What is PyTorch? PyTorch is an open-source machine learning library for Python... [[Wiki](https://en.wikipedia.org/wiki/PyTorch)] - Low-level API - Automatic Differentiation - GPU Acceleration - Dynamic Computation Graph PyTorch e-book - https://pytorch.org/deep-learning-with-pytorch --- ## Outline - Installation - Basic Data Type - Data Loading and Processing - Building Neural Network - Training and Testing --- ## Installation - Install `pytorch` and `torchvision` on official site: https://pytorch.org/get-started/locally/#start-locally  --- ## Before Installing... - Install Python from python.org or Anaconda/Miniconda - Choose a package manager `pip`/`conda` - Install CUDA toolkit (if GPU supports are needed) - NVIDIA Graphics Card --- ## Example Use `conda` as package manager - Run `conda create -n pytorch python=3` to create a virtual environment `pytorch` - Run `source activate pytorch` to activate my virtual environment `pytorch` - Run `conda install pytorch torchvision cudatoolkit=9.0 -c pytorch` to install --- ## Test Installation - Run `python -c 'import torch; print(torch.__version__)'` on Bash ```bash $ python -c 'import torch; print(torch.__version__)' 1.4.0 ``` ## Test CUDA Installation (Test GPU Support) - Run `python -c 'import torch; print(torch.cuda.is_available())'` ```bash $ python -c 'import torch; print(torch.cuda.is_available())' True ``` --- ## Basic Data Type: `torch.Tensor` - **Tensor**: almost equal to multi-dimensional arrays - The **shape** of a tensor: the number of dimensions for each rank - Examples - the shape of a $255 \times 255$ gray scale image is $(255, 255)$ - the shape of a $255 \times 255$ RGB color image is $(3, 255, 255)$, the number of channels is $3$. - the shape of a 10-batch $255 \times 255$ RGB color image is $(10, 3, 255, 255)$ --- [`torch.Tensor`](https://pytorch.org/docs/stable/tensors.html) - Almost equal to `numpy.ndarray` [[NumPy](http://www.numpy.org/)] - Support construct by built-in `list` ```python >>> import torch >>> x = torch.tensor([[5, 4], [8, 7]]) >>> x.shape torch.Size([2, 2]) ``` --- ## Gradient Computation 1. Set `requires_grad=True` in a `torch.Tensor` object 2. Context manager (`with...as...` statement) - `torch.set_grad_enabled(True)` or `torch.set_grad_enabled(False)` - `torch.enable_grad()` - `torch.no_grad()` --- ## An Example of Gradient Computation - $f(x,y)=x^2 + 2y$ ```python >>> def f(x, y): ... return x.pow(2) + 2*y ``` - Set $x=8, y=7$ and `requires_grad=True` if gradient computation is needed ```python >>> x = torch.tensor([8.], requires_grad=True) >>> y = torch.tensor([7.], requires_grad=True) ``` - $f(8, 7)=78$ ```python >>> f(x, y) tensor([78.], grad_fn=<AddBackward0>) ``` $$ \nabla f= \left( \dfrac{\partial f}{\partial x}, \dfrac{\partial f}{\partial y} \right) = \left( 2x, 2 \right) $$ $$ \Rightarrow \nabla f(8, 7)=(16, 2) $$ ```python >>> f(x, y).backward() >>> x tensor([8.], requires_grad=True) >>> x.grad tensor([16.]) >>> y.grad tensor([2.]) ``` --- # GPU Computation - Default device is CPU ```python >>> x = torch.tensor([8.0]) >>> x.device device(type='cpu') ``` - Transfer data to GPU ``` >>> x.to(torch.device('cuda')) tensor([8.], device='cuda:0') ``` - Specify GPU ID ``` >>> x.to(torch.device('cuda:1')) tensor([8.], device='cuda:1') ``` --- ## Data Loading and Processing - `torchvision.transforms` - `torch.utils.data.Dataset` - `torch.utils.data.Dataloader` --- ## `torchvision.transforms` Transforms are common image transformations. They can be chained together using `Compose`. - Transforms on `PIL.Image` - `ColorJitter(brightness=0, contrast=0, saturation=0, hue=0)` - `RandomCrop(size, padding=None, pad_if_needed=False, fill=0, padding_mode='constant')` - `Resize(size, interpolation=2)` - Transforms on `torch.Tensor` - `Normalize(mean, std, inplace=False)` - `RandomErasing(p=0.5, scale=(0.02, 0.33), ratio=(0.3, 3.3), value=0, inplace=False)` - Conversion Transforms - `ToPILImage(mode=None)` - Convert a `torch.Tensor` or an `numpy.ndarray` to `PIL.Image`. - `ToTensor` - Convert a `PIL.Image` or `numpy.ndarray` to `torch.Tensor`. - Functional Transforms --- ## An Example of `torchvision.transforms` ```python= data_transform = transforms.Compose([ transforms.RandomResizedCrop(224), transforms.ToTensor(), transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]) ]) ``` --- ## `torch.utils.data.Dataset` [`torch.utils.data.Dataset`](https://pytorch.org/docs/stable/data.html#torch.utils.data.Dataset) - An abstract class representing a Dataset. To use... 1. derive `torch.utils.data.Dataset` 1. implement - `__len__` for the size of the dataset - `__getitem__` to access the dataset with index --- ## An Example of `torch.utils.data.Dataset` [[Reference](https://github.com/remorsecs/Kaggle-Plant-Seedlings-Classification-Example/blob/master/dataset.py)] - derive `torch.utils.data.Dataset` ```python from torch.utils.data import Dataset class PlantSeedlingDataset(Dataset): def __init__(self, root_dir, transform=None): self.root_dir = Path(root_dir) self.x = [] self.y = [] self.transform = transform self.num_classes = 0 if self.root_dir.name == 'train': for i, _dir in enumerate(self.root_dir.glob('*')): for file in _dir.glob('*'): self.x.append(file) self.y.append(i) self.num_classes += 1 ... ``` [[Reference](https://github.com/remorsecs/Kaggle-Plant-Seedlings-Classification-Example/blob/master/dataset.py)] - Implement `__len__` for the size of the dataset ```python=22 def __len__(self): return len(self.x) ``` - Implement `__getitem__` to access the dataset with index ```python=25 def __getitem__(self, index): image = Image.open(self.x[index]).convert('RGB') if self.transform: image = self.transform(image) return image, self.y[index] ``` --- ## `torch.utils.data.DataLoader` [`torch.utils.data.DataLoader`](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader) - An interface for data loader - Usage ```python torch.utils.data.DataLoader( dataset, batch_size=1, shuffle=False, sampler=None, batch_sampler=None, num_workers=0, ... ) ``` --- ## An Example of `torch.utils.data.DataLoader` [[Reference](https://github.com/remorsecs/Kaggle-Plant-Seedlings-Classification-Example/blob/master/train.py)] ```python train_set = PlantSeedlingDataset( Path(DATASET_ROOT).joinpath('train'), data_transform, ) data_loader = DataLoader( dataset=train_set, batch_size=32, shuffle=True, ) ``` --- ## Building Neural Network To build a neural network, you need - model - loss function - optimizer --- ## Building a Model with `torch.nn.Module` [`torch.nn.Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) - Base class for all neural network modules. To use... 1. derive `torch.nn.Module` 1. implement `forward` 2. call it as function - don't call `forward()` directly --- ## An Example of `torch.nn.Module` [[Reference](https://pytorch.org/docs/stable/nn.html#torch.nn.Module)] ```python import torch.nn as nn import torch.nn.functional as F class Model(nn.Module): def __init__(self): super(Model, self).__init__() self.conv1 = nn.Conv2d(1, 20, 5) self.conv2 = nn.Conv2d(20, 20, 5) def forward(self, x): x = F.relu(self.conv1(x)) return F.relu(self.conv2(x)) model = Model() y = model(x) ``` --- ## Loss Function `torch.nn.*Loss`, e.g. - `torch.nn.L1Loss` - `torch.nn.MSELoss` - ... see: https://pytorch.org/docs/stable/nn.html#loss-functions `torch.nn.functional.*`, e.g. - `torch.nn.functional.binary_cross_entropy` - `torch.nn.functional.binary_cross_entropy_with_logits` - ... see: https://pytorch.org/docs/stable/nn.html#id51 --- ## Optimizer in `torch.optim` `torch.optim` is a package implementing various optimization algorithms. - Stochastic Gradient Descent (SGD) example ```python optimizer = torch.optim.SGD( model.parameters(), lr=0.01, momentum=0.9, ) ``` --- ## Training Loop over the dataset - Fetch data from `dataloader` (training set) - Forward - pass data into model and loss function - Backward - compute the gradient of loss value and use optimizer to update parameters ## Testing Loop over the dataset - Fetch data from `dataloader` (test set) - Forward - pass data into model to get the results - Evaluate --- ## A Simple Example of Training ```python def train(model, device, dataloader, optimizer, num_epochs): for epoch in range(num_epochs): for (inputs, targets) in dataloader: inputs = inputs.to(device) targets = targets.to(device) # forward outputs = model(inputs) loss = F.cross_entropy(outputs, targets) # backward optimizer.zero_grad() loss.backward() optimizer.step() ``` --- ## A Simple Example of Testing ```python def test(model, device, dataloader, optimizer): with torch.no_grad(): for (inputs, targets) in dataloader: inputs = inputs.to(device) targets = targets.to(device) # forward outputs = model(inputs) loss = F.cross_entropy(outputs, targets) # evaluate outputs ``` --- ###### tags: `python` `guide`