# Feature Visualization
###### *Group 16: Wei-Tse Yang, Yifei Liu and Sunwei Wang*

## Introduction
**Feature visualization** can be treated as an optimization problem. Given a neural network using activation optimization we can generate images that activate a certain neuron/channel/layer the most.
In this blogpost, we try to reproduce the results in the [distill post](https://distill.pub/2017/feature-visualization/). The original code used ```Lucid``` library. **We adopt some of the code from the lucid library, but we attempted to implement the procedures on Pytorch.** The goal of this project focuses on how the core idea and techniques of feature visualization can improve the explainability of the convolutional neural network(CNN). The core idea is to obtain the most activated image using activation maximization techniques with the help of some available tricks (such as frequency penalization, transformation, etc.) explained in the blog.

### Motivation
As a user of machine learning models, the human is not just interested in the output and performance of the machine learning model, the human is also intrigued by how a certain prediction is made. which is the basic idea behind explainability, the extra attempt at explaining decisions made by the computer, in a way humans can understand why the computer did what it did.

If the AI system can produce good results, why cannot people just trust the model and the reason why it made the decisions? The problem is that a single metric, such as classification precision or recall, is not a complete description of the most real-world problems. Doshi-Velez and Kim [2017] So according to Molnar [2020], when it is regarding to predictive modeling, there is a trade-off that has to be made: Do the users just want to know **what** is predicted? Or do the users want to know **why** the prediction was made?

### Why the demand for interpretable models

There are several reasons which drive the demand for interpretable models, four major reasons are listed as follows: （Molnar [2020]）

- Human curiosity and learning
- Find meaning in the world
- Detecting bias
- Manage social interactions (Miller [2019])

In 2017, one transparency project, the Defense Advanced Research Projects Agency (DARPA) XAI program was introduced by Gunning [2017], aiming to produce "glass box" models that are explainable/interpretable to human users.

The different interpretation methods can be approximately distinguished according to their results asfollowing: （Molnar [2020]）

- Feature summary statistic
- Feature summary visualization
- Model internals
- Data point
- Intrinsically interpretable model

## Related Work

In this report, we are going to focus on the second interpretation method, our approach is mainly based on [Feature Visualization distill post](https://distill.pub/2017/feature-visualization/). This blogpost discussed the different techniques used to visualize the learned features by activation maximization.

Besides the main paper from (Olah et al. [2017]), which provided a toolbox of current methods to visualize patterns encoded in different convolutional layers of a pre-trained CNN. There are various kinds of visualization techniques that have been developed for feature visualization in a convolutional neural network. The mainstream methods for CNN visualization are the gradient-based methods (Simonyan et al., [2013]; Springenberg et al., [2015]) These methods mainly compute the gradients of the score of a given CNN unit with the respect to the input image. They use gradients to estimate the image appearance that maximizes the unit score.(Zhang&Zhu [2018]) We are also using the gradient-based method which is similar to Olah, but we did not use the toolbox ```Lucid``` library which was introduced by Olah. Instead, we implemented the algorithm ourselves with improvements such as such as frequency penalization and transformation to obtain a better visualization for the images.

## Methods

### Visualization through optimization

Feature visualization can be treated as an optimization problem. CNNs are pre-trained and the weights are fixed. The visualization is the image that can maximize the activation. We use the activation of a unit as an example. We can define the feature visualization with math formula in the following equation (Molnar [2020]). $img∗$ is the optimized image. $img$ is the input image. $h$ is the activation. $x$ and $y$ are the spatial positions of neurons. $n$ is the index of layers. $z$ is the index of channels.

$$img^*=argmax_{img}h_{x,y,z,n}(img)$$

To optimize the equation, we can define the loss function as -1 times the average of the activation. We start from a random noise image. Then, similar to train CNNs, we use the backpropagation and choose the optimizer and learning rate to optimize the image. The whole process is iterative. We show the visualized image obtained through training epochs.

<div style="text-align:center;">
<img src='https://drive.google.com/uc?id=14jZY4IyFu151YMVGyT95yA_H5V4Pk-T5' width="700px"/>
</div>

The advantage of the method is that the visualization does not rely on the input image selected from the dataset. The input is a random noise image. The visualization with optimization generally looks better than those from the deconvolutional network and perturbation-based methods. However, visualization with optimization is sensitive to the learning rate. The visualization would suffer from the high-frequency noise since the gradient is not bounded, shown in the following figure from Olah et al. [2017].

<div style="text-align:center;">
<img src='https://drive.google.com/uc?id=1k8YLYxXDaAsoH5rlBXpn259yLmbnYXmU' width="150px"/>
</div>

### Improvements

Using the "vanilla" version of the algorithm should provide us a corresponding network for the specified layer/channel/neuron. However, without certain modifications, we will only get an image full of noise that the network responds strongly to. Similar to the blog post, we have applied transformation, frequency penalization and upscaling to obtain a better image.

- **Transformation**: The concept beyond the transformation is that the visualization should still activate the model even if the image is slightly changed (Olah et al. [2017]). It's similar to apply the data augmentation for increasing the robustness in CNNs. The random cropping, padding, scaling, and rotation are applied in practice before the input image is fed into the model.
- **Frequency Penalization**: L1 regularization will be added to limit the drastically. The objective then becomes:
\\[img^*= argmax_{img}(h_{x,y,z,n}(img)+ \lambda|img - c| )\\]
Moreover, we blur images in every k steps to remove the high-frequency pattern.

- **Upscaling**: We want the visualization with higher resolution (bigger image sizes) but without the high-frequency pattern. The upscaling method assumes we can learn from the lower-frequency pattern from a lower resolution image (Graetz [2019]). Instead of directly optimizing a high-resolution image, we start from a smaller image with random noise. After optimizing a few steps, we slightly upscale the image. This process is repetitive until we can get the target size. When upscaling the image, we also blur the image to remove the high-frequency noise. The method can make optimization more efficient since we do not directly start from optimizing a big image. Also, the method allows the visualization to easily achieve better resolution.

### Visualization Choices

A network contains multiple neurons, channels and layers. In order to visualize the concepts easier at a higher level. We focus on the visualization of output neurons(before softmax) in this blog. In general, A maximized output neurons means that this input image will be the most probable image for this class.

<!-- - Convolution Neuron: Individual neuron of feature maps in Fig 6 (A).
- Convolution Channel: One channel of feature maps in Fig 6 (B).
- Convolution Layer: The whole channels of feature maps in Fig 6 (C).
- Neuron: One element of the activation for the hully-connected layer in Fig 6 (D).
- Class Logits (Olah et al. [2017]): The last activation before the softmax.
- Class Probability: The last layer for the class prediction in Fig 6 (F).
-->
<!-- <img src='https://drive.google.com/uc?id=11T75yE3rTyMAHmGWBIFOms8of8-lizDO' width="600px"/>
Visualization choices, Molnar [2020]. -->

## Experiment

### Dataset and training details
For our experiment, we will use pretrained ResNet50, Antialiased CNN, and texture CNN as a demonstration in this blog. These networks were trained on the ImageNet dataset. Since that input of our networks is noise images, we will not require any additional image datasets.

For all experiments, we use the ResNet50 as our model. We use Adam as the optimizer and learning rate set to 0.001. The data is generated with random noise and optimized with 4096 steps.

### Strategy for the high-frequency noise
In practice, combining multiple strategies is in general. We planned to conduct an ablation study for each method. However, we found some single methods cannot achieve good quality. Instead of an ablation study, we will show improvement by combing multiple strategies. In this section, we visualize the class logit for Brambling, which is the class index of 10 among 1000 ImageNet classes.

#### Transformation
We show the visualization with the combination of cropping, padding, rotation, and scaling. We set factors with 12 columns/rows in cropping, 24 columns/rows in padding, 20 degrees in rotation, and 1.5 in scaling. More details can be refered to the notebook. The visualization can clearly present the class objects.

<div style="text-align:center;">
<img src='https://drive.google.com/uc?id=1yWYcj0KvDC15-9e7nToKp3R6HbFfPTP1' width="300px"/>
</div>

#### Transformation + Frequency Penalization:
We set $k$=300, $\lambda=0.5$, and $c=0.5$. We show the visualization with the transformation and the frequency penalty as below. The visualization has less noise(gray edge and pink noise) than that of the transformation.
<div style="text-align:center;">
<img src='https://drive.google.com/uc?id=1OA_Fo85-Sm5jSszT0_CGeGdVWCd7OKqz' width="300"/>
</div>

#### Transformation + Upscaling
We show the visualization with the transformation and upscaling as below. The image is upscaled with the factor of 1.1 in every 50 steps.The image size starts from 64x64, and target size is 400x400. The figure is slightly improved with the less gray high-frequency edge. It also has more class objects and texture in the middle of the image.

<div style="text-align:center;">
<img src='https://drive.google.com/uc?id=1t703T5kQR9xd55kDoj0mYJX7L8qBumt8' width="300"/>
</div>

### Top layers, Buttom layers, and Class Logit

We also visualize different channels from the bottom layer to the class logit with the transformation and the upscaling. The Layer $i$ represents the $i$th ResNet block in Pytorch. The first row shows the 256th channel of the Layer 1 on the left side and the 512th channel of Layer 2 on the right side.

<div style="text-align:center;">
<img title="Layer 1" src='https://drive.google.com/uc?id=17us2DoXDXB2hEZHfc6m4mmXxnVpRvJ2z' width="300" />
<img title="Layer 2" src='https://drive.google.com/uc?id=1_K03M47BpSNsAYMdGS659Ft5x1-0o6cC' width="300"/>
</div>

The second row shows the 512th channel of the Layer 3 on the left side and the 512th channel of Layer 4 on the right side.

<div style="text-align:center;">
<img title="Layer 3" src='https://drive.google.com/uc?id=1iP4r0zIG6EUL02pc5H5KAvEY_mDArPpZ' width="300"/>
<img title="Layer 4" src='https://drive.google.com/uc?id=1Xzc7jcO1tubknL-0uc5zOJ68KyO0MaB-' width="300"/>
</div>

The last row shows the visualization of the class logit for Brambling. Generally, the visualizations from the hidden layers become more abstract from the bottom layers to top layers.
<div style="text-align:center;">
<img title="Class Logit" src='https://drive.google.com/uc?id=1t703T5kQR9xd55kDoj0mYJX7L8qBumt8' width="300"/>
</div>

### Further Applications

The methods can be applied to other CNNs research. The visualization might reflect the change when the architecture of CNNs is modified. We pick two papers from the seminar: [Making Convolutional Networks Shift-Invariant Again](https://arxiv.org/abs/1904.11486) and [ImageNet-trained CNNs are biased towards texture](https://openreview.net/forum?id=Bygh9j09KX).

We present the visualizations from the vanilla CNNs and two modified CNNs from the paper. We keep using ResNet50 as the backbone in all three models and applying the transformation and the upscaling during the optimization. We show the class logit of 4 classes as below. The first column is the vanilla model pretrained with ImageNet. The second column is the model from the paper of shift-invariance. The third column is the model from the paper of texture-bias.

**Brambling**
<div style="text-align:center;">
<img title="Ordinary Model" src='https://drive.google.com/uc?id=16-VKOnjfsQ_BwowPH2SLv4m6kh-3CffM' width="200px"/>
<img title="Shift-invariance" src='https://drive.google.com/uc?id=1jge48dyT-GKNaBvOrt3lxUk6s6Vadtqd' width="200px"/>
<img title="Texture-Biased" src='https://drive.google.com/uc?id=1X_1yXZfu09ehYGMmLcIYyKOzsmszYTAx' width="200px"/>
</div>

**Black Swan**
<div style="text-align:center;">
<img title="Ordinary Model" src='https://drive.google.com/uc?id=13eSDImcS8jjAnYfsVKe2GJong59EINyi' width="200px"/>
<img title="Shift-invariance" src='https://drive.google.com/uc?id=1y_PXub0XSeIP24wjvkyI3MOnZTg2zx5l' width="200px"/>
<img title="Texture-Biased" src='https://drive.google.com/uc?id=1kd-rKisqjzAagB9dRk0eKrQ1rl4Kx5A5' width="200px"/>
</div>

**Pizza**
<div style="text-align:center;">
<img title="Ordinary Model" src='https://drive.google.com/uc?id=1kCBIrguX-7g-1Qr9yKXnayUCb4DfqerE' width="200px"/>
<img title="Shift-invariance" src='https://drive.google.com/uc?id=1jkzNUwgYLotIdDnxSPZQDMkqvf-EVlWe' width="200px"/>
<img title="Texture-Biased" src='https://drive.google.com/uc?id=1zmT41yLXI7rsHITq-DNlx8RI-JQ_gpkE' width="200px"/>
</div>

**iPod**
<div style="text-align:center;">
<img title="Ordinary Model" src='https://drive.google.com/uc?id=1RweEFQu6rqdysMx2XIXU_E6YnhmKO3u1' width="200px"/>
<img title="Shift-invariance" src='https://drive.google.com/uc?id=1Gm9oVYIHRhNb4zmENo8kt9eXCmJaKbZL' width="200px"/>
<img title="Texture-Biased" src='https://drive.google.com/uc?id=17Eh3Te9KVeQpLw9q10lFBPhTWTLfCUaM' width="200px"/>
</div>

For the class logit of the iPod, we can see the famous Apple icon. Although it is difficult to tell the difference, the model from the paper of shift-invariance tends to have more class objects overlapped in the image, shown in the following figure.

<div style="text-align:center;">
<img title="Shift-invariance" src='https://drive.google.com/uc?id=1jge48dyT-GKNaBvOrt3lxUk6s6Vadtqd' width="300px"/>
</div>

Also, the visualization from the paper of texture-bias tends to present a clearer shape of objects, such as the bird's head, tree's structure and the edge from the pizza.

<div style="text-align:center;">
<img title="Texture-Biased" src='https://drive.google.com/uc?id=1X_1yXZfu09ehYGMmLcIYyKOzsmszYTAx' width="300px"/>
<img title="Texture-Biased" src='https://drive.google.com/uc?id=1zmT41yLXI7rsHITq-DNlx8RI-JQ_gpkE' width="300px"/>
</div>

### Dicusssion
We discuss the shortcomings in this section. The method is sensitive to the learning rate. Too large learning rates would cause an image full of noise. The optimization can finish within 10 minutes with a 400x400 image. However, the hyperparameters might need to be re-tuned if the visualization choice is changed.

Moreover, not all of the visualization choices would make sense. For some choices in the hidden layers, the visualization might not present a certain shape and pattern. Even if we visualize the class logit, some visualizations only implicitly present corresponding class objects. Also, the method generally has a bad visualization for artificial objects, such as the iPod. Although the evaluation is subjective, the visualization is less natural-looking.

## Conclusion

There have been vast improvements and progress been made by the neural feature visualization community in recent years. There are several challenges such as dealing with high-frequency noises in the field of feature visualization research. This report explored some of the regularization techniques which could be used to reduce the noise and produce better image outputs.

In the journey to make neural networks more explainable/interpretable, feature visualization appears to be one of the most encouraging and well-researched directions. Feature visualization alone does not provide sufficient explainability for the neural networks, but it can be perceived as one of the basic building blocks that, with the help of other techniques or tools, helping people to understand the decisions behind the machine learning models.(Olah et al. [2017])

## Code
Our code can be found in this [Notebook here](https://colab.research.google.com/drive/1mNqZMztQ6YaDTrLUeX857TxDWJN6Oj5i?usp=sharing).

## Reference

D. Bau, B. Zhou, A. Khosla, A. Oliva, and A. Torralba. Network dissection: Quantifying interpretability of deep visual representations. CoRR, abs/1704.05796, 2017. URL http://arxiv.org/abs/1704.05796.

R. Caruana, Y. Lou, J. Gehrke, P. Koch, M. Sturm, and N. Elhadad. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pages 1721–1730, 2015.

D. Gunning. Explainable artificial intelligence (xai). Defense Advanced Research Projects Agency (DARPA), nd Web, 2, 2017.

S. Khademi. Explainability slides from computer vision by deep learning 2019-2020, 2020.

C. Molnar. Interpretable machine learning. Lulu. com, 2020.

C. Olah, A. Mordvintsev, and L. Schubert. Feature visualization. Distill, 2017. doi: 10.23915/distill.00007. https://distill.pub/2017/feature-visualization.

W. Samek, A. Binder, G. Montavon, S. Bach, and K.-R. Müller. Evaluating the visualization of what a deep neural network has learned, 2015.

W. Samek, T. Wiegand, and K.-R. Müller. Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models. arXiv preprint arXiv:1708.08296, 2017.

M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks, 2013.

K. Simonyan, A. Vedaldi, and A. Zisserman. "Deep inside convolutional networks: Visualising image classification models and saliency maps." arXiv preprint arXiv:1312.6034 ,2013.

J.T. Springenberg. "Unsupervised and semi-supervised learning with categorical generative adversarial networks." arXiv preprint arXiv:1511.06390, 2015.

Q.S. Zhang, and S.C. Zhu. Visual interpretability for deep learning: a survey. Frontiers of Information Technology & Electronic Engineering, 19(1), pp.27-39, 2018.