# Week7-3-3

## Denoising autoencoder

Fig.1 shows the manifold of the denoising autoencoder and the intuition of how it works.

<center>
<img src="https://i.imgur.com/nwgxHHJ.png" width="500px" /><br>
<b>Fig. 1</b>: Denoising autoencoder<br>
</center>

In this model we assume we are injecting the same noisy distribution we are going to observer in reality, so that we can learn how to robustly recover from it.
By comparing the input and output, we can tell that the points that already on the manifold data did not move, and the points that far away from the manifold moved a lot.

Fig.2 gives a the relationship between the input data and output data.

<center>
<img src="https://i.imgur.com/uLrm1RA.png" width="350px" />
<img src="https://i.imgur.com/2Kt1yoJ.png" width="330px" />
<br>
<b>Fig. 2</b>: Input and output of denoising autoencoder<br>
</center>

We can also using different colors to represent the distance of each input point moves, Fig.3 shows the diagram.

<center>
<img src="https://i.imgur.com/2RI6n9z.png" width="500px" /><br>
<b>Fig. 3</b>: Measuring the traveling distance of the input data<br>
</center>

The lighter the color, the longer the distance a point traveled. From the diagram, we can tell that the points at the corners traveled close to 1 unit, whereas the points within the 2 branches didn't move at all, since they are attracted by the top and bottom branches during the training process.

## Contractive autoencoder

Fig.4 shows the loss function of the contractive autoencoder and the manifold.

<center>
<img src="https://i.imgur.com/l1HXfz9.png" width="500px" /><br>
<b>Fig. 4</b>: Contractive autoencoder <br>
</center>

The loss function contains the reconstruction term plus squared norm of the gradient of the hidden representation with respect to the input. Therefore, the overall loss will minimize the variation of the hidden layer given variation of the input. The benefit would be to make the model insensitive to reconstruction directions while insensitive to any other possibility directions.

Fig.5 shows how these autoencoders work in general.

<center>
<img src="https://i.imgur.com/zb3mwx2.png" width="500px" /><br>
<b>Fig.5</b>: Basic autoencoder <br>
</center>

The training manifold is a single dimensional object going in three dimensions. Where $x\in X\subseteq\mathbb{R}^{n}$, The goal for autoencoder is to stretch down the curly line in one direction, where $z\in Z\subseteq\mathbb{R}^{d}$ As a result, a point from the input layer will be transformed to a point in the latent layer. Now we have the correspondent between points in the input space and the points on the latent space. But do not have the correspondent between regions of the input space and regions of the latent space. Afterwards, we will utilize the decoder to transform a point from the latent layer to generate a meaningful output layer.

## Implement autoencoder - Notebook

The Jupyter Notebook can be found [here](https://github.com/Atcold/pytorch-Deep-Learning/blob/master/10-autoencoder.ipynb).

In this notebook, we are going to implement a standard autoencoder and a denoising autoencoder and then compare the outputs.

*Define autoencoder model architecture and reconstruction loss*
Using $28 \times 28$ image, and a 30-dimensional hidden layer. The transformation routine would be going from $784\to30\to784$. By applying hyperbolic tangent function to encoder and decoder routine, we are able to limit the output range to $(-1, 1)$. Mean Squared Error (MSE) loss will be used as the loss function of this model.

class Autoencoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(n, d),
            nn.Tanh(),
        )
        self.decoder = nn.Sequential(
            nn.Linear(d, n),
            nn.Tanh(),
        )

    def forward(self, x):
        x = self.encoder(x)
        x = self.decoder(x)
        return x
    
model = Autoencoder().to(device)
criterion = nn.MSELoss()

**Train standard autoencoder**

To train a standard autoencoder using PyTorch, you need put the following 5 methods in the training loop:
#### Going forward:
1) Sending the input image through the model by calling `output = model(img)` .
2) Compute the loss using: `criterion(output, img.data)`.
#### Going backward:
3) Clear the gradient to make sure we do not accumulate the value: `optimizer.zero_grad()`.
4) Back propagation: `loss.backward()`
5) Step backwards: `optimizer.step()`

Fig.6 shows the output of the standard autoencoder.

<center>
<img src="https://i.imgur.com/AtzwFCM.png" width="500px" /><br>
<b>Fig.6</b>: Output of standard autoencoder <br>
</center>

<br />

**Train denoising autoencoder**

For denoising autoencoder, you need to add the following steps:
1) Calling `nn.Dropout()` to randomly turning off neurons.
2) Create noise mask: `do(torch.ones(img.shape))`.
3) Create bad images by multiply good images to the binoary masks: `img_bad = (img * noise).to(device)`.

Fig.7 shows the output of the denoising autoencoder.

<center>
<img src="https://i.imgur.com/X5g5ooX.png" width="500px" /><br>
<b>Fig.7</b>: Output of denoising autoencoder <br>
</center>