# openVINO (**Open** **V**isual **I**nferencing and **N**eural Network **O**ptimization)
---
###### tags: `openVINO` `Docker` `Author:John Chen`
---
## 0. Overview
:::success
Note: [**Reference**](https://www.learnopencv.com/using-openvino-with-opencv/)
:::
OpenVINO的全名是Open Visual Inferencing and Neural Network Optimization，可看出它的目的就是為了加速inference任務而生，它內建的Inference Engine可針對DNN(Deep Neural Network)模型進行最佳化處理，讓Intel系列的CPU (如PC端)、VPU (如Movidius Stick)及ARM CPU (如樹莓派)在無GPU的環境中也能快速的運行。畢竟，GPU相對於CPU昂貴太多了，在Training過程中使用有其必要性，但是眾多且資源貧瘠的inference端就沒有使用GPU的必要，因此，如何在CPU上加速inference model的運行，成了目前AＩ新興的課題。
我們比較不care Training過程所花費的時間，但倒是相當在意Inference所需要的時間。想想看，一個嚴謹的DNN雖需花費一個月才能訓練完成，但輸出的model具有相當效率能在數秒內正確的輸出預測值，相對於一個雖數小時內就能訓練完成、但給出的預測值準確度相當低的模型，你會選擇使用那一個？
因此，在不影響預測準確率的前提下，如何讓Inference用途的model縮小其體積、如何減少模型中用不到的神經元數目、如何加速其執行速度讓預測速度更快，最終讓資源受限的edge端在執行inference時更具效率，這就是目前流行的「quantization」的主要目的。
* 一般進行DNN(Deep Neural Network)的流程有兩大步驟：
1. Training：利用大量的資料，使用Tensorflow, MxNet, Caffe, Keras…等framework，依需求設計出DNN模型，接著透過GPU、CPU、TPU…進行秏時的運算以輸出model及weights，這個過程稱為Training。
2. Inference：取得了model和weights檔，我們便能將testing或actual資料餵給模型得出預測值，這個使用模型來預測資料結果的動作稱為Inference。
* 從上圖中，我們可以看到openVINO所提供最重要的功能是Model Optimizer以及Inference Engine，這是openVINO的動作流程：
1. Train a Model：首先我們如同以往的方式，訓練出一個DL Model。
2. Run Model Optimizer：執行openVINO的模型優化器，將Inference Model進行優化，產出xml及bin兩種檔案，稱為IR(Intermediate Representation)。
3. openVINO的Inference Engine可直接讀取IR model。
4. 使用者可透過openVINO Toolkit與Inference Engine進行溝通整合。
The IR model is hardware agnostic, but OpenVINO optimizes running this model on specific hardware through the Inference Engine plugin. This plugin is available for all intel hardware (GPUs, CPUs, VPUs, FPGAs).
:::success
Note: [**Inference Engine**](https://docs.openvinotoolkit.org/latest/_docs_IE_DG_inference_engine_intro.html)
:::
---
## 1. Optimize Model and create an Intermediate Representation (IR)
:::success
Note: [**Optimization Guide**](https://docs.openvinotoolkit.org/latest/_docs_optimization_guide_dldt_optimization_guide.html)
:::
The model obtained in the previous step is usually not optimized for performance. Therefore, instead of directly using the trained model for inference, OpenVINO requires us to create an optimized model which they call Intermediate Representation (IR) using a Model Optimizer tool they provide. IR is completely hardware agnostic and only depends on the architecture of the neural network.
The figure below shows the difference between OpenVINO deployment method and most other deep learning framework deployment methods.
* The Model Optimizer optimizes the model by the following mechanism
1. Cutting off parts of the model : Removing parts of the network that are required at the time of training, but not at the time of inference. DropOut is an example of such a layer.
2. Fusing operations: Sometimes multiple opertions can be combined into a single operation. The model optimizer detects such patterns and performs the necessary fusion.
* The result of the optimization process is an IR model. The model is split into two files
1. model.xml : This XML file contains the network architecture.
2. model.bin : This binary file contains the weights and biases.
While OpenCV DNN in itself is highly optimized, with the help of Inference Engine we can further increase its performance. The figure below shows the two paths we can take while using OpenCV DNN. We highly recommend using OpenVINO with OpenCV in production when it is available for your platform.
## 2. Download Intel openVINO toolkit
* :open_file_folder: [**Offical openVINO package**](https://software.intel.com/en-us/openvino-toolkit/choose-download/free-download-linux)
**or**
* :open_file_folder: [**Intel Data Base**](https://download.01.org/opencv/2019/openvinotoolkit/R1/)
## 3. Install openVINO
### [**Reference**](https://software.intel.com/en-us/articles/get-started-with-neural-compute-stick)
### 3.1 Install openVION from the Dockerfile (X86 platform)
:::success
Or you can download existing docker image form Docker Hub
```
$ sudo docker pull sunnike/x86_openvino:aewin_20190725
```
:::
```dockerfile=
FROM ubuntu:16.04
ADD l_openvino_toolkit* /openvino/
ARG INSTALL_DIR=/opt/intel/openvino
RUN apt-get update && apt-get -y upgrade && apt-get autoremove
#Install needed dependences
RUN apt-get install -y --no-install-recommends \
build-essential \
cpio \
curl \
git \
lsb-release \
pciutils \
python3.5 \
python3.5-dev \
python3-pip \
python3-setuptools \
udev \
sudo
# installing OpenVINO dependencies
RUN cd /openvino/ && \
./install_openvino_dependencies.sh
RUN pip3 install numpy
# installing OpenVINO itself
RUN cd /openvino/ && \
sed -i 's/decline/accept/g' silent.cfg && \
./install.sh --silent silent.cfg
# Model Optimizer
RUN cd $INSTALL_DIR/deployment_tools/model_optimizer/install_prerequisites && \
./install_prerequisites.sh
# clean up
RUN apt autoremove -y && \
rm -rf /openvino /var/lib/apt/lists/*
RUN /bin/bash -c "source $INSTALL_DIR/bin/setupvars.sh"
RUN echo "source $INSTALL_DIR/bin/setupvars.sh" >> /root/.bashrc
CMD ["/bin/bash"]
```
## 3. openvino-docker
### 3.1 [Reference](https://github.com/mateoguzman/openvino-docker)
:::warning
If you already have the docker image, please go to the 3.3
:::
Intel® OpenVINO™ Toolkit environment
This Dockerfile will provide you with a base environment to run your inference models with OpenVINO™.
### 3.2 Building the Docker Image
#### 3.2.1 Download Intel® OpenVINO™ Toolkit
The firt thing you need is to download the OpenVINO(tm) toolkit.
You can register and download it from the following link (Linux version):
* :open_file_folder: [**Offical openVINO package**](https://software.intel.com/en-us/openvino-toolkit/choose-download/free-download-linux)
**or**
* :open_file_folder: [**Intel Data Base**](https://download.01.org/opencv/2019/openvinotoolkit/R1/)
Or use wget to get the package directly (Latest version is 2019 R1 by the time writing this guide)
```
$ wget http://registrationcenter-download.intel.com/akdlm/irc_nas/15382/l_openvino_toolkit_p_2019.1.144.tgz
```
#### 3.2.2 Extract the file in the root folder
```
$ tar -xf l_openvino_toolkit*
```
#### 3.2.3 Build the image
```
$ docker build -t openvino .
```
### 3.3 Using the image
#### 3.3.1 Run a container
Additionally, for running a sample application that displays an image, you need to share the host display to be accessed from guest Docker container.
The following flags needs to be added to the docker run command:
* --net=host
* --env="DISPLAY"
* --volume="$HOME/.Xauthority:/root/.Xauthority:rw"
To run the docker-app image with the display enabled:
```
$ sudo docker run -v /etc/udev:/etc/udev:rw --privileged -v /dev:/dev:shared --rm --name openvino --net=host --env="DISPLAY" --volume="$HOME/.Xauthority:/root/.Xauthority:rw" -ti sunnike/x86_openvino:aewin_20190725 /bin/bash
```
#### 3.3.2 Run a demo
Once inside the container, go to the Inference Engine demo directory:
```
$ cd /opt/intel/openvino/deployment_tools/demo
$ source ../../bin/setupvars.sh
$ ../../install_NCS_udev_rules.sh
```
Run the inference pipeline demo on NCS2:
```
$ ./demo_security_barrier_camera.sh -d MYRIAD
```
## 4. [Converting a TensorFlow* Model](https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_TensorFlow.html)
:::success
Note: [**Supported Frameworks Layer**](https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_Supported_Frameworks_Layers.html)
:::
### 4.1 [Configure the Model Optimizer](https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_Config_Model_Optimizer.html)
* Using Configuration Scripts
To configure all three frameworks, go to the
==<INSTALL_DIR>/deployment_tools/model_optimizer/install_prerequisites== directory and run:
```bash=
./install_prerequisites.sh
```
* For TensorFlow* on Linux:
```bash=
./install_prerequisites_tf.sh
```
### 4.2 [Freezing Custom Models in Python*](https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_TensorFlow.html#freeze-the-tensorflow-model)
When a network is defined in Python* code, you have to create an inference graph file. Usually graphs are built in a form that allows model training. That means that all trainable parameters are represented as variables in the graph. To be able to use such graph with Model Optimizer such graph should be frozen. The graph is frozen and dumped to a file with the following code:
```python=
import tensorflow as tf
from tensorflow.python.framework import graph_io
frozen = tf.graph_util.convert_variables_to_constants(sess, sess.graph_def, ["name_of_the_output_node"])
graph_io.write_graph(frozen, './', 'inference_graph.pb', as_text=False)
```
Where:
* sess is the instance of the TensorFlow* Session object where the network topology is defined.
* ["name_of_the_output_node"] is the list of output node names in the graph; frozen graph will include only those nodes from the original sess.graph_def that are directly or indirectly used to compute given output nodes. 'name_of_the_output_node' here is an example of possible output node name. You should derive the names based on your own graph.
* ./ is the directory where the inference graph file should be generated.
* inference_graph.pb is the name of the generated inference graph file.
* as_text specifies whether the generated file should be in human readable text format or binary.
### 4.3 [Convert a TensorFlow* Model](https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_TensorFlow.html#Convert_From_TF)
To produce an optimized [**Intermediate Representation (IR)**](https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_convert_model_IRLayersCatalogSpec.html#SoftMax) of the model based on the trained network topology, weights, and biases values.)
* To convert a TensorFlow model:
* Go to the ==<INSTALL_DIR>/deployment_tools/model_optimizer== directory
* Use the `mo_tf.py` script to simply convert a model with the path to the input model .pb file:
```python=
python3 mo_tf.py --input_model <INPUT_MODEL>.pb
```
* Two groups of parameters are available to convert your model:
* [**Framework-agnostic parameters**](https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_convert_model_Converting_Model_General.html): These parameters are used to convert any model trained in any supported framework.
* [**TensorFlow-specific parameters**](https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_TensorFlow.html#tensorflow_specific_conversion_params): Parameters used to convert only TensorFlow models
:::info
**NOTE**: The Model Optimizer does not revert input channels from RGB to BGR by default, as it did in the 2017 R3 Beta release. Manually specify the command-line parameter to perform this reversion: --reverse_input_channels.
:::
:::success
[**Example of convering a Tensorflow model**](https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_convert_model_tf_specific_Convert_Object_Detection_API_Models.html)
:::
* Using TensorFlow*-Specific Conversion Parameters
The following list provides the TensorFlow*-specific parameters.
```
TensorFlow*-specific parameters:
--input_model_is_text
TensorFlow*: treat the input model file as a text
protobuf format. If not specified, the Model Optimizer
treats it as a binary file by default.
--input_checkpoint INPUT_CHECKPOINT
TensorFlow*: variables file to load.
--input_meta_graph INPUT_META_GRAPH
Tensorflow*: a file with a meta-graph of the model
before freezing
--saved_model_dir SAVED_MODEL_DIR
TensorFlow*: directory representing non frozen model
--saved_model_tags SAVED_MODEL_TAGS
Group of tag(s) of the MetaGraphDef to load, in string
format, separated by ','. For tag-set contains
multiple tags, all tags must be passed in.
--offload_unsupported_operations_to_tf
TensorFlow*: automatically offload unsupported
operations to TensorFlow*
--tensorflow_subgraph_patterns TENSORFLOW_SUBGRAPH_PATTERNS
TensorFlow*: a list of comma separated patterns that
will be applied to TensorFlow* node names to infer a
part of the graph using TensorFlow*.
--tensorflow_operation_patterns TENSORFLOW_OPERATION_PATTERNS
TensorFlow*: a list of comma separated patterns that
will be applied to TensorFlow* node type (ops) to
infer these operations using TensorFlow*.
--tensorflow_custom_operations_config_update TENSORFLOW_CUSTOM_OPERATIONS_CONFIG_UPDATE
TensorFlow*: update the configuration file with node
name patterns with input/output nodes information.
--tensorflow_use_custom_operations_config TENSORFLOW_USE_CUSTOM_OPERATIONS_CONFIG
TensorFlow*: use the configuration file with custom
operation description.
--tensorflow_object_detection_api_pipeline_config TENSORFLOW_OBJECT_DETECTION_API_PIPELINE_CONFIG
TensorFlow*: path to the pipeline configuration file
used to generate model created with help of Object
Detection API.
--tensorboard_logdir TENSORBOARD_LOGDIR
TensorFlow*: dump the input graph to a given directory
that should be used with TensorBoard.
--tensorflow_custom_layer_libraries TENSORFLOW_CUSTOM_LAYER_LIBRARIES
TensorFlow*: comma separated list of shared libraries
with TensorFlow* custom operations implementation.
--disable_nhwc_to_nchw
Disables default translation from NHWC to NCHW
```
* Covert Aewin Smoking Detection IR model
```
$ /opt/intel/openvino/deployment_tools/modeloprimizer/mo_tf.py --data_type FP16 --input_model ./frozen_model_inception_resnet_V2_1218.pb --output_dir . --batch 1 --maen (127.5, 127.5, 127.5) --scale 127.5
```
## 5. Demo smoking IR model
```
# Enter the python3.5 virtual environment
$ mavirtualenv openvino -p python3.5
$ cd ~/smoke_model
$ ./demo_video.sh
```
---
# Intel NCSDK Conatainer
## 1. Dockerfile
```dockerfile=
FROM ubuntu:16.04
ARG http_proxy
ARG https_proxy
ENV http_proxy ${http_proxy}
ENV https_proxy ${https_proxy}
RUN echo $https_proxy
RUN echo $http_proxy
# Uncomment the two lines below if you wish to use an Ubuntu mirror repository
# that is closer to you (and hence faster). The 'sources.list' file inside the
# 'tools/docker/' folder is set to use one of Ubuntu's official mirror in Taiwan.
# You should update this file based on your own location. For a list of official
# Ubuntu mirror repositories, check out: https://launchpad.net/ubuntu/+archivemirrors
#COPY sources.list /etc/apt
#RUN rm /var/lib/apt/lists/* -vf
RUN apt-get update \
&& apt-get upgrade -y \
&& apt-get install -y \
build-essential \
git \
lsb-release \
sudo \
udev \
usbutils \
wget \
&& apt-get clean all
RUN useradd -c "Movidius User" -m movidius
COPY 10-installer /etc/sudoers.d/
RUN mkdir -p /etc/udev/rules.d/
USER movidius
WORKDIR /home/movidius
RUN git clone https://github.com/movidius/ncsdk.git
WORKDIR /home/movidius/ncsdk
RUN make install
RUN make examples
```
---
My docker image usage.
```
sudo docker run --name ncsdk_v1 --rm -it --net=host -v /etc/udev:/etc/udev:rw -v /etc/apt/apt.conf:/etc/apt/apt.conf:ro --privileged -v /dev:/dev:shared -v /media/data2/NCS/:/media/data2/NCS/ sunnike/ncsdk:v1 /bin/bash
```
Sign in with Wallet
Connect another wallet