# Notes on "[Data-Free Learning of Student Networks](https://arxiv.org/abs/1904.01186)"

###### tags: `notes` `knowledge-distillation` `adversarial`

Author: [Akshay Kulkarni](https://akshayk07.weebly.com/)

## Brief Outline
- The pre-trained teacher network is considered as a fixed discriminator and a generator generates training samples which can obtain maximum response from the discriminator.
- Simultaneously, a smaller network is trained using the generated data and the teacher network.

## Introduction
- Most existing network compression and speed-up algorithms have a strong assumption that the original training data is available. However, training data may be unavailable due to privacy and transmission limitations.
- Sometimes, the parameters and architecture of pre-trained networks may be unknown, except the input and output layers.
- Thus, conventional methods won't work when either of the above 2 constraints apply.
- Two methods have been proposed before this to work without original training data:
- [Lopes et. al. 2017](https://arxiv.org/abs/1710.07535) utilized 'meta-data' (like means and standard deviations of activations from each layer) recorded from the original training dataset. However, those are not generally provided in most cases.
- [Srinivas and Babu, 2015](https://arxiv.org/abs/1507.06149) compressed the pretrained network by merging similar neurons in fully connected layers.
- The given heavy neural network is regarded as a fixed discriminator.
- Then, a generative network is established for alternating the original training set by extracting information from the network during the adversarial procedure, which can be utlized for learning smaller networks with acceptable performance.

## Methodology

### Teacher-Student Interactions
- Let $\mathcal{N}_T$ and $\mathcal{N}_S$ denote the original pre-trained CNN (teacher) and the desired portable network (student) respectively.
- The student network can be optimized using

$$
\mathcal{L}_{KD} = \frac{1}{n}\sum_i \mathcal{H}_{cross} (y_S^i, y_T^i)
\tag{1}
$$

- Here, $\mathcal{H}_{cross}$ is the cross-entropy loss, $y_T^i=\mathcal{N}_T(x^i)$ and $y_S^i=\mathcal{N}_S(x^i)$ are the outputs of teacher network $\mathcal{N}_T$ and student network $\mathcal{N}_S$, respectively.

### GAN for Generating Training Samples
- They exploit GAN to generate training samples utilizing the available information of the given network (teacher).
- Assuming some familiarity with GANs (if not refer [this blog by Lilian Weng](https://lilianweng.github.io/lil-log/2017/08/20/from-GAN-to-WGAN.html) or [my notes on a review paper on GANs](https://hackmd.io/fWTf_RCBTOKf5IkV6A7Fag)), the loss function for vanilla GANs is:

$$
\mathcal{L}_{GAN} = \mathbb{E}_{y\sim p_{data}(y)}[\log D(y)] + \mathbb{E}_{z\sim p_z(z)}[\log(1 - D(G(z)))]
\tag{2}
$$

- [Odena, 2016](https://arxiv.org/abs/1606.01583) suggested that the tasks of discrimination and classification can improve each other. Instead of training a new $D$, the given teacher network can extract semantic features from images, since it has been trained on large scale datasets.
- However, using the teacher as the discriminator, the output is to classify images to different concept sets, instead of indicating the reality of the images (as in vanilla GANs).
- Thus, the loss function of GANs is inapplicable to approximate the original training set. They conduct analysis on real images and their responses on the teacher network and devise new loss functions based on their observations.

### Training Procedure
#### One-Hot Loss
- Denote the generator and teacher network as $G$ and $\mathcal{N}_T$ respectively. Given a set of random vectors $\{z^1, z^2, \dots, z^n\}$, the images generated from these are $\{x^1, x^2, \dots, x^n\}$ where $x^i=G(z^i)$.
- Inputting these images into $\mathcal{N}_T$, outputs are $\{y_T^1, y_T^2, \dots, y_T^n\}$ with $y_T^i=\mathcal{N}_T(x^i)$. The predicted labels $\{t^1, t^2, \dots, t^n\}$ are calculated (as usual) using $t^i=\arg \max_j(y^i_T)_j$.
- If the images generated by $G$ follow the same distribution as the training data of $\mathcal{N}_T$, they should have similar outputs as the training data.
- Thus, they introduce a *one-hot loss*, which encourages the outputs of generated images by the teacher network to be close to one-hot like vectors. Taking $\{t^1, t^2, \dots, t^n\}$ as pseudo ground truth labels, the one-hot loss is given by

$$
\mathcal{L}_{oh} = \frac{1}{n} \sum_i \mathcal{H}_{cross}(y^i_T, t^i)
\tag{3}
$$

- Here, $\mathcal{H}_{cross}$ is the cross entropy loss function. Using this, they pursue generation of synthetic images exclusively compatible with $\mathcal{N}_T$ instead of general real images.
- Basically, they try to force the outputs of generated images to be closer to one-hot which implies those generated images will be more like the training data (since only the training data type images will get one-hot type output from $\mathcal{N}_T$).

#### Activation Loss
- Features extracted by convolution filters are supposed to contain valuable information about the input images.
- Denote features of $x^i$ extracted by $\mathcal{N}_T$ as $f_T^i$ (the output before the fully-connected layer). Since, the filters in $\mathcal{N}_T$ have been trained to extract intrinsic patterns in the training data, the feature maps tend to receive higher activations if input images are real. Thus, *activation loss* is formulated as

$$
\mathcal{L}_a = -\frac{1}{n}\sum_i ||f_T^i||_1
\tag{4}
$$

- Here, $||.||_1$ is the conventional $l_1$ norm.

#### Information Entropy Loss
- Usually, to ease the training procedure, the number of training examples in each category is balanced. Thus, they employ an information entropy loss to measure the class balance of generated images.
- Given a probability vector $p=(p_1, p_2, \dots, p_k)$, the information entropy (which measures the degree of confusion) of $p$ is calculated as $\mathcal{H}_{info}(p)=-\frac{1}{k}\sum_i p_i \log(p_i)$
- This value indicates the amount of information in $p$ (and is maximum when all variables are $\frac{1}{k}$).
- Given a set of output vectors $\{y^1_T, y^2_T, \dots, y^n_T\}$, where $y_T^i=\mathcal{N}_T(x^i)$, the frequency distribution of generated images for every class is $\frac{1}{n}\sum_i y_T^i$. The *information entropy loss* of generated images is defined as

$$
\mathcal{L}_{ie}=-\mathcal{H}_{info}(\frac{1}{n}\sum_i y_T^i)
\tag{5}
$$

- When the loss is minimized, every element in vector $\frac{1}{n}\sum_i y_T^i$ would be $\frac{1}{k}$, which implies that $G$ could generate images of each category with roughly the same probability (thus leading to a balanced set of synthetic images).

#### Combining the 3 losses
- The final objective function is

$$
\mathcal{L}_{total} = \mathcal{L}_{oh} + \alpha \mathcal{L}_a + \beta \mathcal{L}_{ie}
\tag{6}
$$

- Here, $\alpha$ and $\beta$ are hyperparameters for balancing the three different terms.
- Note that [Simonyan et. al. 2013](https://arxiv.org/abs/1312.6034) and [Mahendran and Vedaldi, 2014](https://arxiv.org/abs/1412.0035) synthesized images by optimizing the input of the NN using backprop. However, it is difficult to generate abundant images since each synthetic image leads to an independent optimization problem solved by backprop.
- In contrast, this method can imitate the training data distribution directly, which is more flexible and efficient to generate new images.


### Optimization
- Training is done in 2 steps as shown in above Algorithm.
- The paper also mentions how gradients for loss functions in Eq. 4 and 5 can be calculated. Both are straightforward derivations (single step, at most two) and can be seen in the paper on page 5 if required (won't be required for implementation based on autograd packages).

## Experiments
### Ablation Experiments
- This table shows the results of the ablation experiments. They suggest that each component of $\mathcal{L}_{total}$ is meaningful.
- By applying the proposed method, $G$ can generate balanced samples from different classes with a similar distribution as that in the original dataset, which is effective for training the student network.

## Conclusion
- This method can perform Knowledge Distillation without detailed architecture information and without training data.
- They present a novel framework to train a generator to approximate the original dataset without training data. Then, a portable network can be learned effectively through the KD scheme.