autoML@mimuw - HackMD

--- title: autoML@mimuw tags: talk description: View the slide with "Slide Mode". type: slide slideOptions: theme: moon font-size: 24px transition: 'fade' center: false ---  ![](https://hackmd.io/_uploads/SkKAtkKEh.png) # autoML: model compression and Neural Architecture Search ---  ## Overview of the talk: 1. What is model compression and how can we achieve it? 2. Model compression approaches. 3. Neural Architecture Search. 4. Our approach at `TCL Research Europe`. ---  ## 2. Model Compression Approaches We can roughly categorize mainstream model compression algorithms as follows: * model quantization * model pruning * low-rank decomposition ----  ## 2.1 Quantization An increasing number of edge devices (smartphones, TVs) are equipped with some form a Neural Processing Unit (NPU), which is a chip designed specifically to handle deep learning models. Running neural networks on these chips is usually much faster and energy-efficient than using edge `GPUs` and `CPUs`. More often than not `NPUs` use integer arithmetic and thus one needs to `quantize` the model before deployment. In principle quantization is not that difficult but once you start to get your hands dirty you immediately notice the following: * you can't just take any of the shelf state-of-the-art model from `HuggingFace` and assume that it can be quantized * NPUs usually support a limited number of operations (forget about quantizing `torch.nn.GeLU`!) * even if you use only supported operations, there is no guarantee that the model you trained will keep its quality after quantization * there are specific architectural patterns in neural networks that will lead to poor quantization and they are quite common * one often has to apply a number of techniques during training like `Quantization-Aware-Training` or other forms of quantization-aware regularizations to minimize the quality drop between a `float` and an `int` model ----  ## 2.2 Pruning It should be no surprise that a large amount of neural networks are heavily overparametrized (look up Lottery Ticket Hypothesis) and their size can be reduced with little quality loss. For example, if you have a detection model trained to provide bounding boxes for 150 classes and your task amounts to detecting just cats and dogs, you probably don't need all the model weights to achieve your objective. In essence there are two types of pruning techniques: * `unstructured pruning` -> pruning weights * `structured pruning` -> pruning channels, attention heads, etc While `unstructured pruning` is probably the best approach from the theoretical perspective (as far as model size reduction goes), it is really hard to make a given model much faster by just removing weights. Few hardware accelerators support sparse models (although there are remarkable attempts by companies like `NeuralMagic` to make it work on cloud `CPUs`). So, one is left with `structured pruning` for all practical purposes. One principle, maybe not intuitive at first glance, that we utilize extensively at `TCL` goes as follows: > If you want to have a small (fast) model, it is usually much better to train a larger model (or take a model pre-trained on some generaic task and plenty of data) first on your task-specific data and then compress it using `structured pruning`, than train a small model from the beginning. ----  ## 2.3 Low-rank decomposition * From linear algebra we know that we can approximate (using Singular Value Decomposition) large matrices $W$ with values in $R^{dim_0 * dim_1}$ with a product of matrices $W_0$ and $W_1$ with values in $R^{dim_0 * r}$ and $R^{r * dim_1}$ ($r$ < $min(dim_0, dim_1)$), respectively. As long as $r$ is sufficiently small, decomposing $W$ will lead to fewer computations. * This can be used to compress both convolutional and linear layers in a neural network. * Given that the input to the layer with $W$ is (after normalization) usually not uniformly distributed in the sphere, it is actually much better to **learn** $W_0$ and $W_1$ using real data. and this can be approached in numerous ways. ---  ## 3. Neural Architecture Search (NAS) NAS algorithms attempt to find neural archi At some point in time (2019-2021) there was a huge number of papers on NAS. Things seem to have slowed down considerably recently From a practically-minded research perspective we can divide `NAS` algorithms into two categories: * **A**: looking for building new architectures from scratch * **B**: taking existing model architectures and trying to make them more efficient on given hardware At `TCL` we mostly focus on **B**, for two reasons: * starting from scratch usually requires a huge amount of compute and does not scale well * having a good starting point that came about after years of research and incorporating plenty of insights seems reasonable --- ## 4. Our approach at `TCL Research Europe`.  Our projects at autoML team can be roughly described as follows: * a number of apps that run on edge devices (smartphones, TVs) require deep learning models to do, e.g, semantic segmentation, object detection, depth estimation and so on * we need these models to run really fast on devices (often in real time -> 30FPS) and achieving that with models/architectures that are available off the shelf (`timm`, `HuggingFace`, official GitHub repositories) is usually not possible * a number of teams in Poland and China need these small and fast models and not everyone is adept at model compression * therefore, we have built in-house expertise in model compression so that all the other teams/projects can ask us to make their solutions more efficient -> this can sometimes mean optimizing the existing model (by, for example, running structured pruning) or downright replacing the model with a new one that is build by us ----  ### 4.1 Model compression in practice - `structured pruning` The paradigm of training a larger model first and then compressing it by removing channels from dense and convolutional layers has proven to be extremely fruitful and allowed us to compress dozens of models for a lot of projects. Although removing channels does not seem to be too complicated at first glance, there are plenty of things one has to get right. In order to achieve that we have developed in-house a number of techniques that, when put together, allow for seamless model compression. Things we did to achieve that include: * designing and implementing an algorithm that will automatically find all the places where the channels in a given model need to be removed "in tandem" -> we handle all models with a single algorithm, there is no need to do manual adjustments * making sure we have a differentiable loss telling us how "big" the model is -> this can be either FLOPs or model latency * handling events like complete removal of specific operations -> it is actually vary common for the algorithm to remove whole branches in the computational graph of the model for a model with a lot of parallel paths <img src="https://hackmd.io/_uploads/SJelUzj82.png" width="600" height="200"> ----  ### 4.2 Model compression in practice - `NAS` As has already been mentioned, we work with NAS algorihms that take existing model architectures (which are quite good to begin with) and modify them by as follows: 1. Divide the model into large, logically connected blocks of operations. We treat these blocks as `teacher blocks` in a blockwise knowledge distillation framework. 2. For each block `b` design a number of diverse or `student blocks`. 3. Train the student blocks in parallel to replicate the outputs of the corresponding teacher blocks. <img src="https://hackmd.io/_uploads/SknySQoUh.png" width="300" height="200"> ----  > It is important that the teachers be diverse in terms hyperparameters like types of layers, depth, width, types of activation functions and so on. 4. Once the student blocks are trained we have a `search space` for model choice - we can pick student blocks to replace their corresponding teachers and, thus, form new architectures. 5. How do we pick the architectures from such a search space? -> We train auxiliary models (usually Graph Neural Networks (GNNs)) to do two things: * provide `embeddings` for student blocks (vector description of student block characteristics) * build `predictors` on top of these `embeddings` to predict metrics like `latency`, `memory consumption`, `accuracy/quality`. The predictors are trained on a small (100-500) number of sampled architectures. 6. With `predictors` we run an evolutionary algorithm to find models that maximize quality given some hardware constraint like latency. The procedure delineated above is an extension and refinement of the paper [DONNA](https://arxiv.org/abs/2012.08859) by Qualcomm Research. ----  Benefits of this approach: * ablations show that given a good pre-trained teacher model, no fine-tuning is necessary for downstream usage -> in other words we just have to train the student blocks to replicate their teachers, which can be done in parallel * we are able to easily switch between, e.g., different activation functions * we can replace blocks that quantize poorly with ones that we know will quantize well * we can explicitly target specific latency on a specific device (the architectures chosen for edge `GPU` will look very different to the ones chosen with edge `CPU` in mind) * we can target metrics like power-consumption as long as we have its reliable measurements ----  ### 4.3 Model compression in practice - `other stuff` Apart from `structured pruning` and the `NAS` approach mentioned before we also focus on other compression techniques: 1. `weighted block pruning` -> a structural pruning technique in which whole blocks of operations (for example an attention block in a Vision Transformer) are removed based on some trade-off between quality and model latency with and without a given block. This may seem a crude approach but is in fact a very versatile tool, since it can: :a: provide a whole family of models (each model with one block less than the previous one) :b: optimized for a specific device. 2. `learned low-rank compression` -> given a linear layer $W$ with values in $R^{dim_0 * dim_1}$ we can decompose it into two linear layers $W_0$ and $W_1$, in $R^{dim_0 * dim}$ and $R^{dim * dim_1}$, respectively, with $dim << dim_0, dim_1$. We do that so that the output of $W_0 * W_1$ matches the output of $W$. Note that the distribution of the input to these layers will, in general NOT be uniform on the unit sphere, so that SVD is not the optimal way to do that.

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.