# 論文 NoteBook - Model Compression

<!-- - 進入 [Book Mode](https://hackmd.io/@johnnyasd12/ryhhmfvim/https%3A%2F%2Fhackmd.io%2Fs%2FBy2jChJk4)

--- -->

<!-- # Survey -->

<!-- ## CS231n - [Lec 15: Efficient Methods and Hardware for Deep Learning](https://hackmd.io/s/By2jChJk4) -->

<!-- ## A Survey of Model Compression and Acceleration for Deep Neural Networks, IEEE Signal Processing Magazine, 2017 -->

<!--

[深度神经网络模型加速与压缩主要方法概述与会议论文列表](https://zhuanlan.zhihu.com/p/43424150)

[Awesome-model-compression-and-acceleration](https://github.com/memoiry/Awesome-model-compression-and-acceleration)

[Awesome Knowledge Distillation](https://github.com/dkozlov/awesome-knowledge-distillation)

[神经网络压缩综述 (論文+概述)](https://blog.csdn.net/qiu931110/article/details/80189905)

[模型压缩那些事（一）](https://zhuanlan.zhihu.com/p/28439056)

[模型压缩总览](https://www.jianshu.com/p/e73851f32c9f)

-->

---

# I. Knowledge Distillation

---

<!-- ## Do Deep Nets Really Need to be Deep?, NIPS'14

---
## Deep Mutual Learning, 2017
- 不需要 teacher model，僅靠 2 個 student 即可相互學習

---

## [[TF](http://github.com/iRapha/replayed_distillation)] Data-Free Knowledge Distillation for Deep Neural Networks, NIPS 2017 Workshop

---

## Large-Scale Domain Adaptation via Teacher-Student Learning, INTERSPEECH 2017

---

## Sarah Tan, Rich Caruana et al., Transparent Model Distillation, 2018

--- -->

## G. Hinton et al. [Distilling the Knowledge in a Neural Network](https://hackmd.io/s/HyofCnRAQ), 2015

### [TTIC Distinguished Lecture Series: Dark Knowledge](https://hackmd.io/s/By1w-0dKX) - Geoffrey Hinton

---
## [FitNets: Hints for Thin Deep Nets](https://hackmd.io/s/S1mVW60RQ), ICLR 2015

---

## [A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning](https://hackmd.io/s/H1-87p0Am), CVPR 2017

---

## [Rocket Launching: A Universal and Efficient Framework for Training Well-performing Light Net](https://hackmd.io/s/HkX4Ep0RX), AAAI 2018

---

## [Learning Efficient Object Detection Models with Knowledge Distillation](https://hackmd.io/s/By47raAR7), NIPS 2017

---
<!-- ## Sequence-Level Knowledge Distillation, EMNLP 2016

---

## Unifying distillation and privileged information

### [GitXiv](http://www.gitxiv.com/posts/wzPhbJgg3BmQjSxzf/unifying-distillation-and-privileged-information)

### 結合兩個東西

---

## MobileID: Face Model Compression by Distilling Knowledge from Neurons, AAAI 2016
---
## Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer, ICLR, 2017

---
## Knowledge Distillation via Generative Adversarial Networks

### [slide](http://aliensunmin.github.io/aii_workshop/2nd/slides/8.pdf)

--- -->

# II. Parameter Quantization

---

## [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://hackmd.io/s/S1BQFpRC7), ICLR 2016 (Best Paper)

---
<!-- ## TOWARDS THE LIMIT OF NETWORK QUANTIZATION, ICLR, 2017

- Hessian Weight Quantization

---

## Loss-aware Binarization of Deep Networks,\[ICLR'17\] cited 30

---
## Deep Learning with Low Precision by Half-wave Gaussian Quantization, \[CVPR'17\] cited 42

---
## Training and inference with integers in deep neural networks\[ICLR'18\] cited 16
--- -->

# III. Parameter Pruning

<!-- ---

## Second order derivatives for network pruning - Optimal brain surgeon, 1993

---
## [X] Data-free parameter pruning for deep neural networks, 2015

--- -->
## [Pytorch] [Learning both Weights and Connections for Efficient Neural Networks, NIPS](https://hackmd.io/s/SJpMgu8gE), 2015

<!-- - [论文 \- Learning both Weights and Connections for Efficient Neural Networks](https://xmfbit.github.io/2018/03/14/paper-network-prune-hansong/)

---

## [X] Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures, 2016

---
## [Caffe] Dynamic Network Surgery for Efficient DNNs, NIPS'16

---
## [Distiller] Learning Structured Sparsity in Deep Neural Networks, NIPS'16, cited 279

--- -->
## [Pytorch] [Pruning Filters for Efficient ConvNets](https://hackmd.io/s/S1rXcT0R7), ICLR 2017

<!-- ---

## [Pytorch] Pruning convolutional neural networks for resource efficient inference, ICLR 2017

---

## [X] Designing Energy-Efficient Convolutional Neural Networks using Energy-Aware Pruning, CVPR 2017

---

## [keras] Soft weight-Sharing for Neural Network Compression, ICLR 2017

---
## [Pytorch] DSD: Dense-Sparse-Dense Training for Deep Neural Networks, ICLR'17
--- -->

## [Caffe] [ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression](https://hackmd.io/s/r1y_JptlN), ICCV 2017

<!-- ---
## [PyTorch] Learning efficient convolutional networks through network slimming. In ICCV, 2017
- [中文筆記](https://blog.csdn.net/u011995719/article/details/78788336)

---
## [Distiller] Stanford University, Google Inc, To prune, or not to prune: exploring the efficacy of pruning for model compression

--- -->
## [Pytorch] [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://hackmd.io/s/HJ7gWUseV), CVPR 2018

<!-- ---
## [PyTorch] Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights

---
## [Tensorflow] Google Brain, Targeted Dropout, NeurIPS 2018 Workshop

- [Hinton 等研究者提出神似剪枝的 Targeted Dropout](https://www.jiqizhixin.com/articles/112501)

---
## [PyTorch] Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers, ICLR 2018
- [GitHub](https://github.com/jack-willturner/batchnorm-pruning)
- [中文筆記](https://blog.csdn.net/hw5226349/article/details/84779325)
- [中文?](https://flashgene.com/archives/157.html)

---
## Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks, IJCAI 2018 -->

# IV. Others

## [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://hackmd.io/s/HJ13qa0CQ), ICLR 2017

---

## Google Inc., [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://hackmd.io/s/BkW7opARm), 2017

<!-- ---

## MobilenetV2: Inverted Residuals and Linear Bottlenecks: Mobile Networks for Classification, Detection and Segmentation

---

## Xception: Deep learning with depthwise separable convolutions

---

# V. Transfer Learning

## [X] An Investigation of Low-Rank Decomposition for Increasing Inference Speed in Deep Neural Networks With Limited Training Data

## [X] Pruning Convolutional Neural Networks for Resource Efficient Transfer Learning

## [X] Transfer Knowledge for High Sparsity in Deep Neural Networks

## [X] Transferring and Compressing CNNs for Face Representation

## [Pytorch] PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning, CVPR 2018

## [X] Pruning Convolutional Filters with First Order Taylor Series Ranking, 2018 -->

<!--

## idea:

桌球
- 看別人怎麼打
- 自己整理出一套思路
- 一直練習
- 比賽、臨場感 (氣氛、模式和平常不一樣)

同學教 rubik's cube
- 自己上網找一些 tutorial
- 花了幾百塊買一顆彈簧式的
- 自己摸索出一些 花式
- 營隊的新朋友也有在玩的，很強；learned

遙控飛機
- 先熟悉電腦模擬的
- 才敢玩真的

tu數學
- 高二 A老師帶我們打比賽
- 高三 B老師教太爛了 他上他的 我讀我的

- 教學相長
- 討論
- 費曼學習法
- teacher-student-2nd_student
- teacher net + student net + auxiliary classifier(with KD)
- 讀書要畫重點
- 課前預習
- 課後複習
- 集中注意力
- 用自己明白的話寫下來
- latent variable 可以重構老師教的內容
- 許多學科都有延續性，若前面單元不求甚解，後面章節就難學得好了
- curriculum learning
- 使用多種學習方法
- 一次只做一件事

xdite
- 快速摘要出幾個關鍵字
- [如何在五天內快速的「學會」一門技能](http://smalltalk.xdite.net/posts/313532-how-to-learn-a-skill-quickly)
- [把一門學問「學好」的紥實方式](http://smalltalk.xdite.net/posts/313535-science-in-a-learning-method-of-solid)

- 把 NN 自己標的 label 也當成部分 label
- 有人做過了
- 上課 學習效果?
- 如何教學?

-->

<!--

Model 從小變大 incrementally 從簡單變複雜

shallow 的 model 從瘦變胖 (neuron 越來越多)，最小化 neuron 數，
Hinton

例如
1. dropout 前面 layer 少，後面 layer 多
可能固定 dropout?
2. Boosting
3. 先學 binary ，後來學 multiclass
4. 二分法，tree，新的樹枝就可以是新的 class?

=========================================
應用

也可做 video highlight 模型簡化

gait 模型簡化

=========================================

可做 LSTM 模型簡化
先學日週期，再學月週期
LSTM簡化不適合 highlight (?

-->

###### tags: `paper notebook` `model compression`