# PyTorch Note
## Tutorial
- https://medium.com/deeplearningbrasilia/deep-learning-introduction-to-pytorch-5bd39421c84
- https://medium.com/dair-ai/a-first-shot-at-deep-learning-with-pytorch-4a8252d30c75
### Defining Your Network
- PyTorch has a standard way for you to create your own models.
- The entire definition should stay inside an object that is a child of the class **nn.Module**.
- Inside this class, there are only two methods that must be implemented.
- `__init__`
- As in other Python class, `__init__` method is used to define the class' attributes and any value that you would want upon instantiation.
- You should always call the `super()` method to initialize the parent class in the PyTorch context.
- Beyond that, you can define all the layers that have optimizable parameters.
- The definition of layers **do not need to be in order** of which they will be used in the network.
- `forward`
- This is the method where you tell how the layers are connected.
- It is worth noticing that there are some other functions applied inside the forward method the `__init__` method but can traditionally be regarded as layers.
- Take a look of instance at the `F.relu` function. We didn't define that in the `__init__` method because it does not have any trainable parameters.
- The following example shows an implementation of a network containing 3 linear layers combined by 2 ReLU layers.

import torch
import torch.nn as nn
import torch.nn.functional as F


class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        # Defining 3 linear layers but NOT the way they should be connected
        # Receives an array of length 240 and outputs one with length 120
        self.fc1 = nn.Linear(240, 120)
        # Receives an array of length 120 and outputs one with length 60
        self.fc2 = nn.Linear(120, 60)
        # Receives an array of length 60 and outputs one with length 10
        self.fc3 = nn.Linear(60, 10)

    def forward(self, x):
        # Defining the way that the layers of the model should be connected
        # Performs RELU on the output of layer 'self.fc1 = nn.Linear(240, 120)'
        x = F.relu(self.fc1(x))
        # Performs RELU on the output of layer 'self.fc2 = nn.Linear(120, 60)'
        x = F.relu(self.fc2(x))
        # Passes the array through the last linear layer 'self.fc3 = nn.Linear(60, 10)'
        x = self.fc3(x)
        return x


net = Net()

- As a rule of thumb, you can put inside the `forward` method all the layers that **do not** have any weights to be updated
- On the other hand, you should put all the layers that have weights to be updated inside the `__init__`.

### Lodaing Your Data: Dataset and Data Loaders
- **Dataset** and **Data loaders** are the tools in PyTorch can define how to access your data.
- For instance, if you have several images in some directory structure, you can personalize the way you access it with the **Dataset** class.
- The code below give a basic example of how to define your own **Dataset** class and then loop over your data accessing the elements.

import torch
import pandas as pd
from torch.utils.data import Dataset, DataLoader

class ExampleDataset(Dataset):
    """Example Dataset"""

    def __init__(self, csv_file):
        """ 
        csv_file (string): Path to the csv file containing data.
        """
        self.data_frame = pd.read_csv(csv_file)

    def __len__(self):
        return len(self.data_frame)

    def __getitem__(self, idx):
        return self.data_frame[idx]

# instantiates the dataset  
example_dataset = ExampleDataset('my_data_file.csv')

# batch size: number of samples returned per iteration
# shuffle: Flag to shuffle the data before reading so you don't read always in the same order
# num_workers: used to load the data in parallel
example_data_loader = DataLoader(example_dataset, , batch_size=4, shuffle=True, num_workers=4)

# Loops over the data 4 samples at a time
for batch_index, batch in enumerate(example_data_loader):
    print(batch_index, batch)
  

- There are 3 required methods we need to impement in the **Dataset** classes:
- `__init__`
- You should put your directories information and other things that would allow to access it in the initialization.
- Inside the `__init__` you do not have to already load your data.
- `__len__`
- In this method, you should implement a way to get the entire size of your dataset.
- For example, you have to implement an approach of counting the total number of files that makes your data if you have a set of images in some directories.
- Nevertheless, this can be arbitraily complicated depending on how your data is stored.
- `__getitem__`
- This is where you implement how to get a single item from your dataset.
- For instance, if you have plenty of images, here is where you would load your image from disk into memory and serve it as what the method returns.
- Using **DataLoader** class results in more efficiency of accessing your dataset (it can read a batch of data in parallel).

### Training: Updating the Network Weights
- optimizer
- It goes over all your weights and update them of you.
- It automatically accesses all the layers your defined in the `__init__` method of your network class and updates them using a certain learning rate and algorithm depending on the kind of optimizer you choose.
- It must have a criterion to use of optimization.
- loss function
- sample code

import torch.optim as optim
import torch.nn as nn
# instantiate your network that should be defined by you
net = Net()
# create your optimizer
optimizer = optim.SGD(net.parameters(), lr=0.01)
# define your criterion for optimization
criterion = nn.MSELoss()
# dat_set comes from somewhere
for data in data_set:
  # zero the gradient buffers
  optimizer.zero_grad()  
  # Passes the data through your network
  output = net.forward(data)
  # calculates the loss
  loss = criterion(output, target)
  # Propagates the loss back
  loss.backward()
  # Updates all the weights of the network
  optimizer.step()    

## Example
### Drawing Architecture: Building Deep Convolutional GAN’s In Pytorch
- https://towardsdatascience.com/drawing-architecure-building-deep-convolutional-gans-in-pytorch-5ed60348d43c
#### Generator

nn.Sequential(
 nn.ConvTranspose2d( nz, ngf * 8, 4, 1, 0, bias=False),
 nn.BatchNorm2d(ngf * 8),
 nn.ReLU(True),
 nn.ConvTranspose2d(ngf * 8, ngf * 4, 4, 2, 1, bias=False),
 nn.BatchNorm2d(ngf * 4),
 nn.ReLU(True),
 nn.ConvTranspose2d( ngf * 4, ngf * 2, 4, 2, 1, bias=False),
 nn.BatchNorm2d(ngf * 2),
 nn.ReLU(True),
 nn.ConvTranspose2d( ngf * 2, ngf, 4, 2, 1, bias=False),
 nn.BatchNorm2d(ngf),
 nn.ReLU(True),
 nn.ConvTranspose2d( ngf, nc, 4, 2, 1, bias=False),
 nn.Tanh()
 )

#### Random Noise

fixed_noise = torch.randn(64, nz, 1, 1, device=device)

#### Discriminator

nn.Sequential(
 nn.Conv2d(nc, ndf, 4, 2, 1, bias=False),
 nn.LeakyReLU(0.2, inplace=True),
 nn.Conv2d(ndf, ndf * 2, 4, 2, 1, bias=False),
 nn.BatchNorm2d(ndf * 2),
 nn.LeakyReLU(0.2, inplace=True),
 nn.Conv2d(ndf * 2, ndf * 4, 4, 2, 1, bias=False),
 nn.BatchNorm2d(ndf * 4),
 nn.LeakyReLU(0.2, inplace=True),
 nn.Conv2d(ndf * 4, ndf * 8, 4, 2, 1, bias=False),
 nn.BatchNorm2d(ndf * 8),
 nn.LeakyReLU(0.2, inplace=True),
 nn.Conv2d(ndf * 8, 1, 4, 1, 0, bias=False),
 nn.Sigmoid()
 )

## Transfer Learning
- https://heartbeat.fritz.ai/transfer-learning-with-pytorch-cfcb69016c72
### Import

%matplotlib inline
%config InlineBackend.figure_format = 'retina'
from matplotlib import pyplot as plt
import torch
from torch import nn
import torch.nn.functional as F
from torch import optim
from torch.autograd import Variable
from torchvision import datasets, transforms, models
from PIL import Image
import numpy as np
import os
from torch.utils.data.sampler import SubsetRandomSampler
import pandas as pd

### Visualizing

img_dir='/content/drive/My Drive/app/cell_images'
def imshow(image):
    """Display image"""
    plt.figure(figsize=(6, 6))
    plt.imshow(image)
    plt.axis('off')
    plt.show()
# Example image
x = Image.open(img_dir + '/Parasitized/C33P1thinF_IMG_20150619_114756a_cell_179.png')
np.array(x).shape
imshow(x)

### Transform

# Define your transforms for the training, validation, and testing sets
train_transforms = transforms.Compose([transforms.RandomResizedCrop(size=256, scale=(0.8, 1.0)),
                                      transforms.RandomRotation(degrees=15),
                                      transforms.ColorJitter(),
                                      transforms.RandomHorizontalFlip(),
                                      transforms.CenterCrop(size=224),  # Image net standards
                                      transforms.ToTensor(),
                                      transforms.Normalize([0.485, 0.456, 0.406],
                                                           [0.229, 0.224, 0.225])
                                      ])

test_transforms = transforms.Compose([transforms.Resize(256),
                                      transforms.CenterCrop(224),
                                      transforms.ToTensor(),
                                      transforms.Normalize([0.485, 0.456, 0.406], 
                                                           [0.229, 0.224, 0.225])])

validation_transforms = transforms.Compose([transforms.Resize(256),
                                            transforms.CenterCrop(224),
                                            transforms.ToTensor(),
                                            transforms.Normalize([0.485, 0.456, 0.406], 
                                                                 [0.229, 0.224, 0.225])])

### Loading Data

#Loading in the dataset

train_data = datasets.ImageFolder(img_dir,transform=train_transforms)
# number of subprocesses to use for data loading
num_workers = 0
# percentage of training set to use as validation
valid_size = 0.2

test_size = 0.1

# obtain training indices that will be used for validation
num_train = len(train_data)
indices = list(range(num_train))
np.random.shuffle(indices)
valid_split = int(np.floor((valid_size) * num_train))
test_split = int(np.floor((valid_size+test_size) * num_train))
valid_idx, test_idx, train_idx = indices[:valid_split], indices[valid_split:test_split], indices[test_split:]

print(len(valid_idx), len(test_idx), len(train_idx))

# define samplers for obtaining training and validation batches
train_sampler = SubsetRandomSampler(train_idx)
valid_sampler = SubsetRandomSampler(valid_idx)
test_sampler = SubsetRandomSampler(test_idx)

# prepare data loaders (combine dataset and sampler)
train_loader = torch.utils.data.DataLoader(train_data, batch_size=32,
    sampler=train_sampler, num_workers=num_workers)
valid_loader = torch.utils.data.DataLoader(train_data, batch_size=32, 
    sampler=valid_sampler, num_workers=num_workers)
test_loader = torch.utils.data.DataLoader(train_data, batch_size=32, 
    sampler=test_sampler, num_workers=num_workers)

### Steps of Training the mode
1. Loading in the pre-trained mode.
2. Freezing the convolutional layers.
3. Replacing the fully connected layers with a custom classifier.
4. Training the custom classifier for the specific task.

#### Loading the Pre-Trained Model

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
#pretrained=True will download a pretrained network for us
model = models.densenet121(pretrained=True)
model

#### Freezing the convolutional layers & replacing the fully connected layers with a custom classifier

#Freezing model parameters and defining the fully connected network to be attached to the model, loss function and the optimizer.
#We there after put the model on the GPUs
for param in model.parameters():
  param.require_grad = False
fc = nn.Sequential(
    nn.Linear(1024, 460),
    nn.ReLU(),
    nn.Dropout(0.4),
    
    nn.Linear(460,2),
    nn.LogSoftmax(dim=1)
    
)
model.classifier = fc
criterion = nn.NLLLoss()
#Over here we want to only update the parameters of the classifier so
optimizer = torch.optim.Adam(model.classifier.parameters(), lr=0.003)
model.to(device)

#### Training the custom classifier for the specific task

#Training the model and saving checkpoints of best performances. That is lower validation loss and higher accuracy
epochs = 10
valid_loss_min = np.Inf
import time
for epoch in range(epochs):
    
    start = time.time()
    
    #scheduler.step()
    model.train()
    
    train_loss = 0.0
    valid_loss = 0.0
    
    for inputs, labels in train_loader:
        
       
        # Move input and label tensors to the default device
        inputs, labels = inputs.to(device), labels.to(device)
        
        optimizer.zero_grad()
        
        logps = model(inputs)
        loss = criterion(logps, labels)
        loss.backward()
        optimizer.step()
train_loss += loss.item()
        
       
    model.eval()
    
    with torch.no_grad():
        accuracy = 0
        for inputs, labels in valid_loader:
            
            inputs, labels = inputs.to(device), labels.to(device)
            logps = model.forward(inputs)
            batch_loss = criterion(logps, labels)
valid_loss += batch_loss.item()
# Calculate accuracy
            ps = torch.exp(logps)
            top_p, top_class = ps.topk(1, dim=1)
            equals = top_class == labels.view(*top_class.shape)
            accuracy += torch.mean(equals.type(torch.FloatTensor)).item()
                    
            
            
# calculate average losses
    train_loss = train_loss/len(train_loader)
    valid_loss = valid_loss/len(valid_loader)
    valid_accuracy = accuracy/len(valid_loader) 
      
    # print training/validation statistics 
    print('Epoch: {} \tTraining Loss: {:.6f} \tValidation Loss: {:.6f} \tValidation Accuracy: {:.6f}'.format(
        epoch + 1, train_loss, valid_loss, valid_accuracy))
            
    
    
    if valid_loss <= valid_loss_min:
      
      
      print('Validation loss decreased ({:.6f} --> {:.6f}).  Saving model ...'.format(
      valid_loss_min,
      valid_loss))
      model_save_name = "Malaria.pt"
      path = F"/content/drive/My Drive/{model_save_name}"
torch.save(model.state_dict(), path)
      valid_loss_min = valid_loss        
       
    print(f"Time per epoch: {(time.time() - start):.3f} seconds")

### Loading a Saved Model from Disk

model.load_state_dict(torch.load('Malaria.pt'))

### Test

def test(model, criterion):
# monitor test loss and accuracy
    test_loss = 0.
    correct = 0.
    total = 0.
for batch_idx, (data, target) in enumerate(test_loader):
        # move to GPU
        if torch.cuda.is_available():
            data, target = data.cuda(), target.cuda()
        # forward pass: compute predicted outputs by passing inputs to the model
        output = model(data)
        # calculate the loss
        loss = criterion(output, target)
        # update average test loss 
        test_loss = test_loss + ((1 / (batch_idx + 1)) * (loss.data - test_loss))
        # convert output probabilities to predicted class
        pred = output.data.max(1, keepdim=True)[1]
        # compare predictions to true label
        correct += np.sum(np.squeeze(pred.eq(target.data.view_as(pred))).cpu().numpy())
        total += data.size(0)
            
    print('Test Loss: {:.6f}\n'.format(test_loss))
print('\nTest Accuracy: %2d%% (%2d/%2d)' % (
        100. * correct / total, correct, total))
test(model, criterion)
Test Loss: 0.257728   Test Accuracy: 90% (2483/2756)

### Predict and Visualize

def load_input_image(img_path):    
    image = Image.open(img_path)
    prediction_transform = transforms.Compose([transforms.Resize(size=(224, 224)),
                                     transforms.ToTensor(), 
                                     transforms.Normalize([0.485, 0.456, 0.406], 
                                                          [0.229, 0.224, 0.225])])

    # discard the transparent, alpha channel (that's the :3) and add the batch dimension
    image = prediction_transform(image)[:3,:,:].unsqueeze(0)
    return image
    
def predict_malaria(model, class_names, img_path):
    # load the image and return the predicted breed
    img = load_input_image(img_path)
    model = model.cpu()
    model.eval()
    idx = torch.argmax(model(img))
    return class_names[idx]
    
from glob import glob
from PIL import Image
from termcolor import colored

class_names=['Parasitized','Uninfected']
inf = np.array(glob(img_dir + "/Parasitized/*"))
uninf = np.array(glob(img_dir + "/Uninfected/*"))
for i in range(3):
    img_path=inf[i]
    img = Image.open(img_path)
    if predict_malaria(model, class_names, img_path) == 'Parasitized':
        print(colored('Parasitized', 'green'))
    else:
        print(colored('Uninfected', 'red'))
    plt.imshow(img)
    plt.show()
for i in range(3):
    img_path=uninf[i]
    img = Image.open(img_path)
    if predict_malaria(model, class_names, img_path) == 'Uninfected':
        print(colored('Uninfected', 'green'))
    else:
        print(colored('Parasitized', 'red'))        
    plt.imshow(img)
    plt.show()

## Image Augmentation
- https://medium.com/analytics-vidhya/image-augmentation-for-deep-learning-using-pytorch-feature-engineering-for-images-3f4a64122614