---
lang-ref: ch.08-3
title: Generative Models - Autoencoders
lecturer: Alfredo Canziani
authors: Nandhitha Raghuram, Xinyi Zhao
date: 25 March 2021
---

## Recap: Predictive Latent Variable Model

As we know latent variables are useful in providing additional information that help in predicting multiple values of the target $\boldsymbol{y}$, assiociated with the input $x$. We have seen earlier, in order to predict the target variable $\boldsymbol{y}$ from latent variable $z$, the energy function is given as,

$$
E(\boldsymbol{y},z) = [y_1 - g_1(z)]^2 + [y_2 - g_2(z)]^2, \boldsymbol{y} \in \boldsymbol{Y}
$$

To be clear, we show the unit of $\boldsymbol{g}$:

$$
\boldsymbol{g} = [g_{1}, g_{2}]^\top, \mathbb{R} \rightarrow \mathbb{R}^{2} \\
\boldsymbol{g}(z) = [w_{1}\cos(z), w_{2}\sin(z)]
$$

In this case, the the latent variable $z$ is one dimensional. However, if the $z$ has a higher dimension, this could lead to an energy function of $0$ everywhere. Hence, in order to ensure lower energy values for more compatible predictions we introduce a regularized factor.

### Training recap

Given an observation $\boldsymbol{y}$, training the regualarized latent variable model

$$
E(\boldsymbol{y},z) = C(\boldsymbol{y},\boldsymbol{\tilde{y}}) + R(z)
$$

where $\boldsymbol{\tilde{y}} = \text{Dec}(z)$,

- compute

$$
F_{\beta}(\boldsymbol{y}) = \underset{z}{\text{softmin}_{\beta}}[E(\boldsymbol{y},z)]
$$

- Minimise

$$
\mathcal{L}(F_{\beta}(\cdot), Y)
$$

At zero temperature limit,
- Compute

$$
\check{z} = \underset{z}{\arg\min}{E(\boldsymbol{y},z)} \\
F_{\infty}(\boldsymbol{y}) = \underset{z}{\min}{E(\boldsymbol{y},\check{z})}
$$

- Minimise

$$
\mathcal{L}(F_{\infty}(\cdot), Y)
$$

### Target propagation

<center>
<img src="{{site.baseurl}}/images/week08/08-3/target_prop.png" style="background-color:#DCDCDC;" /><br>
<b>Fig. 1:</b> Target propagation
</center>

*Fig. 1* shows the architecture of target propagation. Given an observation $\boldsymbol{y}$, we first compute $\tilde{z}$ which is the initial guess of the latent variable $z$

$$
\tilde{z} = \text{Enc}(\boldsymbol{y})
$$

We then initialize $z$ to $\tilde{z}$, here we must ensure that these values are not too far apart from each other.

$$
\check{z} = \underset{z}{\arg\min}{E(\boldsymbol{y},z)} \\
E(\boldsymbol{y},z) = C(\boldsymbol{y}, \boldsymbol{\tilde{y}}) + R(z) + D(z,\tilde{z})
$$

We now minimise the loss functional in two steps, minimise $\mathcal{L}(F_{\infty}(\cdot), Y)$:

$$
\theta_{\text{Dec}} = \theta_{\text{Dec}} - \triangledown_{\theta_{\text{Dec}}}C(\boldsymbol{y},\boldsymbol{\tilde{y}})\\ \\
\theta_{\text{Enc}} = \theta_{\text{Enc}} - \triangledown_{\theta_{\text{Enc}}}D(\check{z},\tilde{z})
$$

In this way $z$ is constrained in taking only a set of values.

## Autoencoder (AE)

An autoencoder is a type of artificial neural network used to learn efficient data codings in an unsupervised manner. The aim is to first learn encoded representations of our data and then generate the input data (as closely as possible) from the learned encoded representations. Thus, the output of an autoencoder is its prediction for the input.

<center>
<img src="{{site.baseurl}}/images/week08/08-3/AE.png" style="background-color:#DCDCDC;" /><br>
<b>Fig. 2</b>: Architecture of a basic autoencoder
</center>

*Fig. 2* shows the architecture of a basic unconditional autoencoder. To summarize at a high level, a very simple form of AE is as follows:

- First, the autoencoder takes in an input and maps it to a hidden state through an affine transformation $\boldsymbol{h} = f(\boldsymbol{W}_h \boldsymbol{y} + \boldsymbol{b}_h)$, where $f$ is an (element-wise) activation function. This is the **encoder** stage. Note that $\boldsymbol{h}$ is also called the **code**.
- Next, $\tilde{\boldsymbol{y}} = g(\boldsymbol{W}_x \boldsymbol{y} + \boldsymbol{b}_y)$, where $g$ is an activation function. This is the **decoder** stage.
- Finally, the energy is the sum of the reconstruction and the regularization, $F(\boldsymbol{y}) = C(\boldsymbol{y},\tilde{\boldsymbol{y}})+ R(\boldsymbol{h})$.

### Reconstruction costs

When the input is real-valued we use mean squared error loss,

$$
C(\boldsymbol{y},\boldsymbol{\tilde{y}}) = \Vert \boldsymbol{y} - \boldsymbol{\tilde{y}} \Vert^{2} = \Vert{\boldsymbol{y} - \text{Dec}[\text{Enc}(\boldsymbol{y})]}\Vert^{2}
$$

When the input is categorical we use cross entropy loss,

$$
C(\boldsymbol{y},\boldsymbol{\tilde{y}}) = -\sum_{i = 1}^{n} [y_{i} \log(\tilde{y_{i}}) + (1 - y_{i})\log(1 - \tilde{y_{i}})]
$$

### Loss functional

The loss functional is the average of the per sample loss.

$$
\mathcal{L}(F(\cdot), Y) = \frac{1}{m} \sum_{j = 1}^{m} \ell(F(\cdot),y^{(j)}) \in \mathbb{R} \\
\ell_\text{energy}(F(\cdot),\boldsymbol{y}) = F(\boldsymbol{y})
$$

### Under-/over-complete hidden layer

When the dimensionality of the hidden layer is less than the dimensionality of the input then it is under complete hidden layer. When the dimensionality of the hidden layer is greater than the dimensionality of the input then it is over complete.

The dimensionality of the hidden layer is larger than the input so that they can be linearly seperable. However, this can lead to a collapsed representation. Since by reconstructing the input, the model can copy all the features. In order to avoid this there are a few techniques such as denoising autoencoder, contractive autoencoder, variation autoencoder, etc.

## Denoising Autoencoder

Denoising Autoencoder is a contrastive technique. *Fig. 3* below shows the architecture and manifold of the denoising autoencoder and the intuition of how it works.

<center>
<img src="{{site.baseurl}}/images/week08/08-3/DAE.png" style="background-color:#DCDCDC;" /><br>
<b>Fig. 3</b>: Denoising Autoencoder
</center>

In this model, we corrupt the input data <span style="color:#80b1d3">$\boldsymbol{y}$</span>, in blue, because it's cold / low energy, and get the sample <span style="color:#fb8072">$\boldsymbol{\hat{y}}$</span>, in red, because it's hot / high energy. The energy associated to $\boldsymbol{\tilde y}$ is squared euclidian distance from its original value. Then $\boldsymbol{\tilde y}$ is encoded back to the hidden variable $h$ and then passed back to the decoder producing $\boldsymbol{\tilde y}$ which should be close to the target <span style="color:#80b1d3">$\boldsymbol{y}$</span>. Hence irrespective of what noise is added to the original data, the autoencoder needs to learn to separate the noise and retrieve the original value of <span style="color:#80b1d3">$\boldsymbol{y}$</span>.

## Contractive Autoencoder

A contractive autoencoder is an unsupervised deep learning technique that helps a neural network encode unlabeled training data. It makes this encoding less sensitive to small variations in its training dataset. This is accomplished by adding a regularizer, or penalty term, to whatever cost or objective function the algorithm is trying to minimize. The end result is to reduce the learned representation’s sensitivity towards the training input.

<center>
<img src="{{site.baseurl}}/images/week08/08-3/contractiveAE.png" style="background-color:#DCDCDC;" /><br>
<b>Fig. 4</b>: Contactive autoencoder
</center>

As shown in *Fig. 4*, $C(\boldsymbol{y}, \boldsymbol{\tilde y})$ penalises sensitivities to reconstructions along the manifold while $R(\boldsymbol{h})$ penalises how much the wiggling of $\boldsymbol{h}$ comes from the wiggling of $\boldsymbol{y}$. In this manner, you basically get to push up the energies only in the direction that are not used for the reconstruction.

## Variational Autoencoder (VAE)

**Intuition behind VAE and a comparison with classic autoencoders**

Next, we introduce Variational Autoencoders (or VAE), a type of generative model. But why do we even care about generative models? To answer the question, discriminative models learn to make predictions given some observations, but generative models aim to simulate the data generation process. One effect is that generative models can better understand the underlying causal relations which leads to better generalization.

### What's the difference between variational autencoder (VAE) and basic autoencoder (AE)?

Note that although VAE has "Autoencoders" (AE) in its name (because of structural or architectural similarity to auto-encoders), the formulations between VAEs and AEs are very different. *Fig. 5* below shows the architectures of VAE and basic AE.

<center>
<img src="{{site.baseurl}}/images/week08/08-3/VAE.png" style="background-color:#DCDCDC;" /><br>
<b>Fig. 5</b>: VAE vs. Basic AE
</center>

For VAE:
- First, the encoder stage: we pass the input $\boldsymbol{y}$ to the encoder. Instead of generating a hidden representation $\boldsymbol{h}$ in AE, the hidden representation in VAE comprises two parts: $\boldsymbol{\mu}$ and $\boldsymbol{v}$. And similar to using regularisation factor $R$ for $\boldsymbol{h}$, we now use regularisation factors $U$ and $V$ for $\boldsymbol{\mu}$ and $\boldsymbol{v}$, respectively.
- Next, we use a sampler to sample $\boldsymbol{z}$ which is the latent random variable following a Gaussian distribution with $\boldsymbol{\mu}$ and $\boldsymbol{v}$. Note that people use Gaussian distributions as the encoded distribution in practice, but other distributions can be used as well.
- Finally, $\boldsymbol{z}$ is passed into the decoder to generate $\tilde{\boldsymbol{y}}$.
- The decoder will be a function from $\mathcal{Z}$ to $\mathbb{R}^{n}$: $\boldsymbol{z} \mapsto \boldsymbol{\tilde{y}}$.

In fact, for basic autoencoder, we can think of $\boldsymbol{h}$ as just the vector $\boldsymbol{\mu}$ in the VAE formulation, with the variance set to zero. In short, the main difference between VAEs and AEs is that VAEs have a good latent space that enables generative process.

### What's the difference between variational autencoder (VAE) and denoising autoencoder (DAE)?

Now let's have compare the difference between VAE and DAE. See *Fig. 6* below.

<center>
<img src="{{site.baseurl}}/images/week08/08-3/VAE_DAE.png" style="background-color:#DCDCDC;" /><br>
<b>Fig. 6</b>: VAE vs. Denoising AE
</center>

For DAE as we mentioned before, the sampling was happening between <span style="color:#80b1d3">$\boldsymbol{y}$</span>, in blue, because it’s cold / low energy, and <span style="color:#fb8072">$\boldsymbol{\hat{y}}$</span>, in red, because it's hot / high energy. So we move the input and then we decode everything into $\boldsymbol{\tilde{y}}$. For VAD we encode the input and we add noise in the hidden , so we just switch the position of the encoder in the sampler more or less.

### The VAE objective (loss) function

<center>
<img src="{{site.baseurl}}/images/week08/08-3/VAEloss.png" style="background-color:#DCDCDC;" /><br>
<b>Fig. 7</b>: Mapping from input space to latent space
</center>

First, we encode from input space (left) to latent space (right), through encoder and noise. Next, we decode from latent space (right) to output space (left). To go from the latent to input space (the generative process) we will need to either learn the distribution (of the latent code) or enforce some structure. In our case, VAE enforces some structure to the latent space.

As usual, to train VAE, we minimize a loss function. The loss function is therefore composed of a reconstruction term as well as a regularization term.
- The reconstruction term is on the final layer (left side of the figure). This corresponds to $C(\boldsymbol{y}, \tilde{\boldsymbol{y}})$ in the figure.
- The regularization term is on the latent layer, to enforce some specific Gaussian structure on the latent space (right side of the figure). We do so by using a penalty term $D_{KL}(\mathcal{N}(\boldsymbol{\mu}, \boldsymbol{v}) \mathrel{\Vert} \mathcal{N}(\boldsymbol{0}, \boldsymbol{1}))$. Without this term, VAE will act like a basic autoencoder, which may lead to overfitting, and we won't have the generative properties that we desire.

### Discussion on sampling $\boldsymbol{z}$ (reparameterisation trick)

How do we add this noise as mensioned above? We here introduce *reparameterization trick*. From above we know that, we sample from the Gaussian distribution, in order to obtain latent variable $\boldsymbol{z}$. However, this is problematic, because when we do gradient descent to train the VAE model, we don't know how to do backpropagation through the sampling module.

We can simply say that the new latent $\boldsymbol{z}$ is going to be $\boldsymbol{\mu}$ which is the guess for the mean plus a epsilon that was sampled from a gaussian whose amplitude is changed by the square root of the the standard deviation $\boldsymbol{v}$. So in this way, we can get gradients back flowing in these encoder.

### Visualizing Latent Variable Estimates and Reconstruction Loss

<center>
<img src="{{site.baseurl}}/images/week08/08-3/bubbles_z.png" style="background-color:#DCDCDC;" /><br>
<b>Fig. 8</b>: Visualizing vector $z$ as bubbles in the latent space
</center>

*In Fig. 8 above, each bubble represents an estimated region of $\boldsymbol{z}$, and the arrows represent how the reconstruction term pushes each estimated value away from the others, which is explained more below.*

As stated above, the Free Energy for the VAE contains two terms: a reconstruction term and a regularization term. We can write this as

$$
\tilde{F}(\boldsymbol{y}) = C(\boldsymbol{y}, \tilde{\boldsymbol{y}}) + \beta D_{\text{KL}}(\mathcal{N}(\boldsymbol{\mu}, \boldsymbol{v}) \mathrel{\Vert} \mathcal{N}(\boldsymbol{0}, \boldsymbol{1}))
$$

To visualize the purpose of each term in the free energy, we can think of each estimated $\boldsymbol{z}$ value as a circle in $2d$ space, where the centre of the circle is $\boldsymbol{\mu}$ and the surrounding area are the possible values of $\boldsymbol{z}$ determined by $\boldsymbol{v}$.

If there is an overlap between any two estimates of $z$, (visually, if two bubbles overlap) this creates ambiguity for reconstruction because the points in the overlap can be mapped to both original inputs. The first term $C(\boldsymbol{y}, \tilde{\boldsymbol{y}})$ forces to reconstructing these bubbles to the correct locations.
- The reconstruction loss $C(\boldsymbol{y}, \tilde{\boldsymbol{y}})$ will push the points away from one another such there is no overlap.
- Another option for $C(\boldsymbol{y}, \tilde{\boldsymbol{y}})$ to avoid overlap is to make the variance zero and then there are just points other than bubbles so there's no more overlap.

However, if we use just the reconstruction loss, the estimates will continue to be pushed away from each other and the system could blow up. This is where the penalty term comes in.

**The penalty term**

The second term is the relative entropy (a measure of the distance between two distributions) between a Gaussian with mean $\boldsymbol{\mu}$, variance $\boldsymbol{\mu}$ and the standard normal distribution. If we expand this second term in the VAE loss function we get:

$$
\beta D_{\text{KL}}(\mathcal{N}(\boldsymbol{\mu}, \boldsymbol{v}) \mathrel{\Vert} \mathcal{N}(\boldsymbol{0}, \boldsymbol{1})) = \frac{\beta}{2} \sum\limits_{i=1}^dv_i - \log{(v_i)} - 1 + \mu_i^2
$$

where the expression in the summation has four terms. Let's use $V$ for the first three terms.

$$
V = v_i - \log{(v_i)} - 1
$$

From the left bottom plot in Fig. 8 above, we can see that this expression is minimized when $v_i$ is 1. Therefore our penalty loss will keep the variance of our estimated latent variables at around 1. Visually, this means our "bubbles" from above will have a radius of around 1.

The last term, $\mu_i^2$, minimizes the distance between the bubbles and therefore prevents the "exploding" encouraged by the reconstruction term.

**Note:** The $\beta$ in the VAE loss function is a hyperparameter that dictates how to weight the reconstruction and penalty terms.