---
# System prepended metadata

title: Neural Pruning with Taylor Expansion (1)
tags: [Taylor Expansion, Pytorch, Neural Pruning, Deep Learning]

---

# 大綱
神經裁減（Neural Pruning）是深度學習領域中的一個重要議題，它通常是指在神經網絡中去除一些不必要的、冗餘的神經元或連結，以簡化模型結構並提高計算效率。這樣的優化過程有助於減少模型的存儲需求、提高推理速度，並降低在硬體上運行模型的能源消耗。

而在神經裁減的其中一塊領域中，利用了taylor expansion去估計神經元被裁減之後損失函數的變化。

在論文"PRUNING CONVOLUTIONAL NEURAL NETWORKS
FOR RESOURCE EFFICIENT INFERENCE"中使用taylor expansion的第1項估計  
$$f(x+h)\approx f(x)+f'(x)\cdot h$$
去估計當神經元(neuron)被裁減(prune)之後，損失函數的變化。將損失函數變化極小對應的神經元裁減掉，就可以達到神經元裁減的目標。

# 符號定義
* train set: $$D=\{(x_1,y_1),(x_2,y_2),\cdots\}$$
* weight of model: $$\Theta=\{W_1^1,W_1^2,\cdots\},$$ $W_i^j=(w_i^j,b_i^j),$代表第$i$層layer的第$j$個神經元權重
* Loss function: $$L(\Theta,D)$$

# 理論推導
* 當第$j$層layer的第$i$個神經元被裁減之後，代表$\exists W^j_i\in\Theta, W^j_i=0$，假設新的weight:
$$\begin{split}\bar{\Theta}=\{W_1^1,W_1^2,\cdots,W_i^j=0,\cdots\}\end{split}$$
* 這時候的損失函數可以用taylor expansion估計成  
$$\begin{split}L(\bar\Theta,D)\approx L(\Theta,D)+\nabla_{W_i^j}L(\Theta,D)\cdot W_i^j\end{split}$$
在這邊$\nabla _{W_i^j}L$代表$L$對 $W_i^j=(w_i^j,b_i^j)$中2個矩陣所有元素偏微分的梯度。

* 那麼損失函數的變化就可以被估計成  
$$|\nabla _{W_i^j}L\cdot W_i^j|\approx \left|L(\Theta,D)-L(\bar\Theta,D)\right|$$
理論上只要對應比較小的$|\nabla _{W_i^j}L\cdot W_i^j|$之神經元拿掉，那麼$L$的值不會有太大的變化。
# Pytorch example

* 以下程式碼在估計某個layer中所有神經元的$|\nabla _{W_i^j}L\cdot W_i^j|$：
```python=
def cal_est_delta_loss(model, data_loader, loss_func, layer_tobe_pruned):

    weight, bias = layer_tobe_pruned.weight, layer_tobe_pruned.bias
    delta_losses = [0 for _ in range(weight.shape[0])]

    for (x, y) in data_loader:
        ybar = model(x)
        loss = loss_func(ybar, y)
        weight_grad, bias_grad = torch.autograd.grad(loss, [weight, bias])
        for i in range(weight.shape[0]):
            delta_losses[i] += (weight_grad[i]*weight[i] + bias_grad[i]*bias[i]).sum().abs().item()

    
    return delta_losses
```

# 結果
以下面這個model在mnist手寫辨識問題上
```python=
class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()

        self.conv1 = nn.Conv2d(in_channels=1, out_channels=8, kernel_size=3, padding=1)
        self.conv2 = nn.Conv2d(in_channels=8, out_channels=16, kernel_size=3, padding=1)
        self.conv3 = nn.Conv2d(in_channels=16, out_channels=32, kernel_size=3, padding=1)
        self.conv4 = nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3, padding=1)
        self.linear1 = nn.Linear(in_features=64*7*7, out_features=128)
        self.linear2 = nn.Linear(in_features=128, out_features=10)

    def forward(self, x):
        size = x.size(0)
        x = F.relu(self.conv1(x))
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = F.relu(self.conv3(x))
        x = F.max_pool2d(F.relu(self.conv4(x)), 2)
        x = x.view(size, -1)
        x = F.relu(self.linear1(x))
        x = F.softmax(self.linear2(x), dim=1)
        return x
```
計算每個神經元的$\displaystyle\left|\frac{\partial L}{\partial W_i^j}W_i^j\right|$，並由小到大將對應的神經元裁減掉以得到以下的結果：
```
Before pruning: loss: 1.480422, acc: 98.0900%, num_parameter: 427210
After pruning 216 neurons: loss: 1.480414, acc: 98.0900%, num_parameter: 6951
After pruning 217 neurons: loss: 1.480417, acc: 98.0900%, num_parameter: 6548
After pruning 218 neurons: loss: 1.480423, acc: 98.0900%, num_parameter: 6145
After pruning 219 neurons: loss: 1.480416, acc: 98.0900%, num_parameter: 5742
After pruning 220 neurons: loss: 1.480416, acc: 98.0900%, num_parameter: 5651
After pruning 221 neurons: loss: 1.480420, acc: 98.0900%, num_parameter: 5248
After pruning 222 neurons: loss: 1.480409, acc: 98.0900%, num_parameter: 4845
After pruning 223 neurons: loss: 1.480410, acc: 98.0900%, num_parameter: 4269
After pruning 224 neurons: loss: 1.480389, acc: 98.0900%, num_parameter: 3915
After pruning 225 neurons: loss: 1.479798, acc: 98.1300%, num_parameter: 3388
After pruning 226 neurons: loss: 1.575055, acc: 88.6100%, num_parameter: 3083
After pruning 227 neurons: loss: 1.659003, acc: 80.1400%, num_parameter: 2778
After pruning 228 neurons: loss: 1.769887, acc: 69.0000%, num_parameter: 2473
After pruning 229 neurons: loss: 1.864964, acc: 59.3100%, num_parameter: 2168
```
只要3388個參數就可以有高達98%以上的準確率。

下一篇將討論taylor expansion二次項估計的應用。
# 參考連結
> - Pruning Convolutional Neural Networks for Resource Efficient Inference, https://arxiv.org/abs/1611.06440