---
//type: slide
slideOptions:
transition: slide
transition-speed: fast

###### tags: `COSCUP`, `ONNC`, `NVDLA`
---

[TOC]

---

# When <span><!-- .element: class="fragment highlight-blue" -->ONNC</span> meets <span><!-- .element: class="fragment highlight-green" -->NVDLA</span>
當開源專案交會時互放的光亮

a127a127 - 蔡德育
[e-mail]

---

## 講者背景
- IOI / ICPC
演算法
- Linker / Loader
Compiler 相關 Toolchain
- JVM / Clang / LLVM
Process VM / Compiler

<!-- .slide: data-background="https://intern-md.skymizer.com/uploads/upload_3c8c34d502c5d84c4a2d11553c256e8e.png"-->

---

## <span style="color:lime">NVDLA</span>

<span>**NV**IDIA **D**eep **L**earning **A**ccelerator
<!-- .element: class="fragment" data-fragment-index="1" -->
</span>

<span>[http://nvdla.org/](http://nvdla.org/)
<!-- .element: class="fragment" data-fragment-index="2" -->
</span>

---

### 什麼是 DLA？

**D**eep **L**earning **A**ccelerator

---

### 加速這種鬼東西的硬體

---

### [N][C][H][W]


---

## NVDLA 詳細是什麼?

> The NVIDIA Deep Learning Accelerator (NVDLA) is a <span><!-- .element: class="fragment highlight-red" -->free and open</span> architecture that promotes a standard way to design deep learning inference accelerators. With its modular architecture, NVDLA is <span><!-- .element: class="fragment highlight-red" -->scalable, highly configurable</span>, and designed to simplify integration and portability.
<!-- .element: class="fragment" data-fragment-index="1" -->

---

> Delivered as an <span style="color:red">open source</span> project under the NVIDIA Open NVDLA License, all of the software, hardware, and documentation will be available on GitHub.

https://github.com/nvdla/

---

<!-- .slide: data-transition="slide-in fade-out" -->


---

<!-- .slide: data-transition="fade-in" -->


---

### CONV

<span>
<!-- .element: class="fragment" data-fragment-index="1" -->
</span>

<span>https://media.giphy.com/media/i4NjAwytgIRDW/giphy.gif
<!-- .element: class="fragment" data-fragment-index="2" -->
</span>

<span>http://cs231n.github.io/convolutional-networks/
<!-- .element: class="fragment" data-fragment-index="3" -->
</span>

---

### SDP

> The <span><!-- .element: class="fragment highlight-red" -->Single Data</span> Point Processor (SDP) allows for the application of both linear and non-linear functions onto individual data points.

簡單的說，就是對個別 data 做一些運算

---

### PDP

> <span><!-- .element: class="fragment highlight-red" -->Planar Data</span> Processor

簡單的說，就是對個別 channel 做一些運算
目前用來做 pooling layer

---

### CDP

> <span><!-- .element: class="fragment highlight-red" -->Cross-channel Data</span> Processor

簡單的說，就是用來做單一 pixel 跨 channel 的運算

目前用來做 LRN，在 channel 方向的 normalization

---

### RUBIK


> The data reshape engine performs <span><!-- .element: class="fragment highlight-red" -->data format transformations</span>

簡單的說，就是用來改變 data layout

---

### BDMA

> The bridge DMA (BDMA) module provides a <span><!-- .element: class="fragment highlight-red" -->data copy</span> engine to move data between the system DRAM and the dedicated high-performance memory interface

簡單的說，就是在 DRAM 和專屬記憶體之間搬資料

---

from https://software.intel.com/en-us/articles/intel-processors-for-deep-learning-training

---

## NVDLA 的 projects

- Verilog<span style="font-size: 0.6em"><!-- .element: class="fragment" data-fragment-index="1" --> - https://github.com/nvdla/hw/tree/nvdlav1/vmod</span>
- C-model<span style="font-size: 0.6em"><!-- .element: class="fragment" data-fragment-index="2" --> - https://github.com/nvdla/hw/tree/nvdlav1/cmod</span>
- Linux drivers<span style="font-size: 0.6em"><!-- .element: class="fragment" data-fragment-index="3" --> - https://github.com/nvdla/sw/tree/master/kmd</span>
- test benches and test suites<span style="font-size: 0.6em"><!-- .element: class="fragment" data-fragment-index="4" --> - https://github.com/nvdla/sw/tree/master/regression</span>
- kernel- and user-mode software<span style="font-size: 0.6em"><!-- .element: class="fragment" data-fragment-index="5" --> - https://github.com/nvdla/sw</span>
- software development tools
- Virtual Platform<span style="font-size: 0.6em"><!-- .element: class="fragment" data-fragment-index="6" --> - https://github.com/nvdla/vp</span>
- Compiler (nvdla_compiler)<span style="font-size: 0.6em"><!-- .element: class="fragment" data-fragment-index="7" --> - 沒有 open source :scream:</span>

---

## <span style="color:DodgerBlue">ONNC</span>
**O**pen **N**eural **N**etwork **C**ompiler
https://onnc.ai/

> ONNC (Open Neural Network Compiler) is a collection of open source, modular, and reusable <span><!-- .element: class="fragment highlight-red" -->compiler</span> algorithms/toolchains <span><!-- .element: class="fragment highlight-red" -->targeted on deep learning accelerators (DLAs)</span>

---

### 什麼是 Compiler？

> A compiler is a computer program that <span><!-- .element: class="fragment highlight-red" -->translates</span> computer code written in <span><!-- .element: class="fragment highlight-red" -->one programming language</span> (the source language) <span><!-- .element: class="fragment highlight-red" -->into another programming language</span> (the target language)

from https://en.wikipedia.org/wiki/Compiler

---

#### Compiler 通常是從高階語言轉到低階的語言

---

#### 以 IR (intermediate representation) 為中心
通常可以分為三個部分
- Front-end
- Source language -> IR
- Middle-end
- IR -> IR
- Back-end
- IR -> Target language

---

#### 其中最核心的就是 middle-end
一個 IR -> IR 的 function，通常在 Compiler 領域叫做 `Pass`，在一個 compiler 中會有非常多的 passes

---

### ONNX
> **O**pen **N**eural **N**etwork e**X**change format

https://onnx.ai/

---

#### 有了 ONNX，front-end 就搞定了


---

### ONNC 還有很多自己的 Middle-end pass

---

#### Backend 很難實作 Gemm？

<span>我們有個 Pass 會幫你把 Gemm 轉成 Conv
<!-- .element: class="fragment" data-fragment-index="1" -->
</span>

<span>從此你只需要實作 Conv 就好
<!-- .element: class="fragment" data-fragment-index="2" -->
</span>

---

#### Conv 在硬體上有 size 限制？

<span>我們有個 Pass 會幫你把 Conv 切小
<!-- .element: class="fragment" data-fragment-index="1" -->
</span>

---


---

### ONNC 的 NVDLA Back-end

<span>就是從 ONNC IR 轉成 NVDLA Loadable file
<!-- .element: class="fragment" data-fragment-index="1" -->
</span>

---

## ONNC 如何幫助 NVDLA

---

### Case 1: Support `GlobalAveragePool`

---

#### `AveragePool`


---

#### NVDLA PDP


> Max, min, and mean pooling methods are supported.

---

#### `GlobalAveragePool`

---

#### 把 kernel 的寬高設成跟 input 一樣就好了


---

#### 但是 ...

---

NVDLA 的 PDP

**不支援** kernel 的高或寬 > 8 的 Pooling layer


---

#### 解法


---

#### 但是 ... ...

---

input 的寬高可能無法被順利的分解

例如他是個質數


---

#### 利用 padding 功能


---

### Case 2: Support Concat

---

The inception module in the inception network.

---

#### 就在 channel 方向把資料依序接起來就好

---

#### 但是 ... ... ...

---

#### 其實在 NVDLA 中 memory layout 不是 NCHW


---

#### 而是 NCHWc， [N][C][H][W] -> [N][C/c][H][W][c]


---

目前官方 compiler 的 Concat 實作，並沒有考慮到在 channel 方向會有 bubble，所以會造成拿到錯誤的值

---

ONNC 可以在發現 Concat 會有 bubble 時，增加新的 Hardware operation 去移除那些 bubble


---

## ONNC + NVDLA Supported Models in ONNX Model Zoo
| | |
|:--------------:|:--------------:|
| AlexNet | GoogLeNet |
| CaffeNet | R-CNN ILSVRC13 |
| DenseNet-121 | Inception v1 |
| Inception v2 | ResNet-50 |
| ShuffleNet | SqueezeNet |
| VGG-19 | ZFNet-512 |

---

## NVDLA 如何幫助 ONNC

<span style="font-size:3em">OPEN SOURCE !!
<!-- .element: class="fragment" data-fragment-index="1" -->
</span>

---

#### 但是 ... ... ... ...


---

### Open Source 之外

---

### 還需要有好的 Document

---


---


---


---


---

## <span style="color:DodgerBlue">ONNC</span> + <span style="color:Lime">NVDLA</span>

<span>可以自行 support <span style="color:DodgerBlue">ONNC</span> 不 support 的 layer
<!-- .element: class="fragment" data-fragment-index="1" -->
</span>

<span>甚至可以自行 support <span style="color:Lime">NVDLA</span> 不 support 的 layer
<!-- .element: class="fragment" data-fragment-index="2" -->
</span>

---

### Relu6

<span>`min(max(features, 0), 6)`
<!-- .element: class="fragment" data-fragment-index="1" -->
</span>

<span>`min(Relu(features), 6)`
<!-- .element: class="fragment" data-fragment-index="2" -->
</span>

---


---

### ROI Pooling

<span>Patches are welcome!
<!-- .element: class="fragment" data-fragment-index="1" -->
</span>

---

## Q & A
<span>
<span style="position:relative;top:-4em;right:-5em;">住在 RD 部門的貓 https://www.facebook.com/hachu.cat/</span>
<!-- .element: class="fragment" data-fragment-index="1" -->
</span>

---

有更多的問題想要私下問，可以寄信到
[e-mail]

願意投履歷的，可以寄信到
[e-mail]