---
# System prepended metadata

title: AI 影像辨識
tags: [AIOT, IOT, AI, Jetson Xavier NX, MQTT]

---

AI 影像辨識
==
###### tags: `AI`,`IOT`,`AIOT`,`MQTT`,`Jetson Xavier NX`

## 1.硬體說明
## 2.作業系統安裝
1.Jetson Xavier NX 作業系統(Linux)建立,請參閱NVIDIA Jetson Nano安裝教學:
>https://d246810g2000.medium.com/nvidia-jetson-nano-for-jetpack-4-4-01-%E7%92%B0%E5%A2%83%E5%AE%89%E8%A3%9D-fd48d5658a13<br>
>[name=張銘][time=Mon, Jul 7, 2020]

2.更新相關套件程式<br>
```python=
sudo apt-get update
sudo apt-get upgrade -y
sudo apt-get full-upgrade
sudo apt-get install python3-pip
python3 -m pip install --upgrade pip
sudo apt-get install build-essential
sudo apt-get install git
sudo apt install curl
sudo -H pip3 install jetson-stats
sudo systemctl restart jetson_stats.service
```    
設定 CUDA 環境變數
```python=
# 安裝 vi 編輯器
sudo apt-get install vi
# 進入 .bashrc 設定
sudo vi ~/.bashrc

加入以下指令
export CUDA_HOME=/usr/local/cuda-10.2
export LD_LIBRARY_PATH=/usr/local/cuda-10.2/lib64:$LD_LIBRARY_PATH
export PATH=/usr/local/cuda-10.2/bin:$PATH

source ~/.bashrc
# 查看是否設定成功
nvcc -V
```  

3.加入硬體M2 NVMe SSD 固態硬碟取代SD卡讀取，以其加快系統存取速度

```python=
$ git clone https://github.com/jetsonhacks/rootOnNVMe
$ cd rootOnNVMe              #進入資料夾
$ ./copy-rootfs-ssd.sh       #SD卡的rootfs複製到SSD
$ ./setup-service.sh         #將root移至SSD
$ sudo reboot                #完成系統重開機
```  
4.更新相關套件程式
```python=
sudo apt-get install python-pip python-dev
sudo pip install --upgrade pip
sudo pip install --upgrade virtualenv
sudo apt-get install cmake libpython3-dev python3-numpy
```
5.兩者存取速度之比較
| 硬體說明 | Average Rate | Average Access Time |
| -------- | ------------ | ------------------- |
| NVMe SSD (128GB) |2832 MB/s      | 0.07ms  |
| SD卡 (64GB)     | 85.2 MB/s  | 0.63ms      |

6.python版本默認是python2.7，切换到python3.6版本
```python=
sudo update-alternatives --install /usr/bin/python python /usr/bin/python2 100
sudo update-alternatives --install /usr/bin/python python /usr/bin/python3 150

sudo update-alternatives --config python
```
選取0,將預設模式設定於Python3

![](https://lh3.googleusercontent.com/pw/AMWts8B2x2J4Wje26gnAwHG2h07_k7E4UPcQu-PcsZO5FaGIc0Bj5PudAu4oG7b2h1ntws-k1IYyPwtP8A2EaXiB8BzCaDC-xVedczi4wPO8XcvpNTxc_eRPgCiOwtHwaal46uglk_lMrTpFhf54D2OuA1oy=w611-h160-s-no?authuser=0)

7.安裝Framework
```python=
########## PyTorch ##########
sudo apt-get install libopenblas-base libopenmpi-dev libomp-dev python3-pip
pip3 install Cython
wget https://nvidia.box.com/shared/static/9eptse6jyly1ggt9axbja2yrmj6pbarc.whl -O  torch-1.6.0-cp36-cp36m-linux_aarch64.whl
pip3 install torch-1.6.0-cp36-cp36m-linux_aarch64.whl
```
安裝完確認PyTorch版本
![](https://lh3.googleusercontent.com/pw/AMWts8AR46ZwF_rek21ndeFVpkpwTqeHFBFEYjWEF85KzTVlnWbArj7l_PjWt0swQKQNtWIfD7DjfgRZVHYcU32zioajxAxGc9vxU7yocXJpgnr1qlupVSyGPVJGJjRdgIoPiAYZb5yg9KiJEO4Ge2lhfZfBZBejjk5Du2bZZdZH1t9OuuZ13j3wmfckiaS8xf72zH2jSCHJCUrqPYFiPyAc3zpISTBogoi9a7vg5IhJ4J7BoVnxkyFjdt76AsAnuUkTv9oGRCxcpnyxx4wjQ3j1hr3AJQoMonT3Aa5kfYtnhLhHo_FYk3qRTr9r00HIdagXC2_0aeBMz3_fnKjKSOBTOccyNFMr9vcvbWK9OG8mriNvOGfe8lW4Ji0uMRH8KehnnNsTjDjTHrw9q5SX1Yrtt6pGYxgwYdc5NmjB_vMxsj52KzdGJxxQR2tg3-W7SG1vWAB4tDDKPWrDo2xlq9qPYZEZtN80k34ZSPI9pzBRfBkwD06R_G3pizSOkPS1Krn_TmI9B-yCcwhiqbrWL3iZaSOp2lZqoCjDkO7c98uXDqF7kjdDmVMhNV1UdN1PKqrdlKMTkvnN9RYj2sl8Nv4cYARdxSoZQSJwgl9DLL1vf_woJDsmVa1gqoFd0QLNNPZLP0g32MxL5rEMO_-5oBsc9I2i5YOsQQO5LChMxtvqLTXnaAmHlUfdD5ZA7XoaktVM2_74uioeAp0-o1tq1kzdndY6uf1epktxVVd_on5i3zKRf_RvmMcOfCkxvC7Rp1imHHkPUkg_JE1Fg4AtcnAo6eO9mqxBGSiyneEjWaRr-h60-cPYpstelt1QSrGmuUOkwneABcxd_Xan1giAMaqNMIkv-bKbSwgegHg_3BG_o2fZVRghxfYkVp3sw2Noqy6B9MZSKFocXdHL_o4kGhA8KpuYMQ=w566-h128-s-no?authuser=0)

8.安裝ONNX Runtime 
```python=
# Download pip wheel from location mentioned above
wget https://nvidia.box.com/shared/static/jy7nqva7l88mq9i8bw3g3sklzf4kccn2.whl -O onnxruntime_gpu-1.10.0-cp36-cp36m-linux_aarch64.whl
# Install pip wheel
pip3 install onnxruntime_gpu-1.10.0-cp36-cp36m-linux_aarch64.whl
```
安裝後確認運行狀況

![image alt](https://lh3.googleusercontent.com/pw/AMWts8BG8xDyOkDQ3JDfwwpjGmwVGXR1CEBA1lPqiKo0-bQa8QvNG2nitnFXImFTP3e2OEzF7ZqVFnNHJzEdxIxnQVh7CMSL23hth-ov2dDEOakw_h_OiF2oX0KYVuoj71iYpBDUxHvIOP19W5MsQylGsWpBemZkjRuHkRcaK16WW5yWDdUbUDni7PpsDbt1FwOOzwNXYuAbO7cgkvam8jhjnBMqK2YQf32D1EhbFCeBw3wKa3laiPNVtl1mGN59oFzUaBTljbpQW6Z8L96zCjpNTaVgoWofwOdJG8kB4DKELRsO5yxoAuBzck-FJodfDnfJT_-XBaMZgVDwn9RWy0r5bCT4hHM7a1FJnKBKRL8nIJRiZH2gmInw5K83KMRpob80dZgoFfm5KxRSmb_DCt-3TqqNtXZrMYhoodF3cEP4wyLy9gTuIHBSz7JFBfj0VcZhIRTushWaBbuA3k0HR9L5F8Sswbbvr5Mh-q3kyNvY6EzfP1WaRwKpnzyrNYlVo6wjJsjFNh-DaKsVkMrkFwnZfyI5OIlH_ICapIIFp3SW6-7WZc1_FVDg6qzMSgbZJsLlPrGyqpbO6BUjlK4frtT6S5Fw2v5-bZE8g6r7yIKKNmW3RDISmDKBE6QDIKq0Bq6VimgdwbcXfWGOpLBmD1F7PltJSi4llWHkb0TbMtxgMusJKDby6Y5GZXA2ECtI89Dpq1A4SgRFX6AP7LgbJ8E8gqqbCCv0FwNBm1AwWFguISkGl22u8IG8Usujs5CUScy3FH1hoafcjwdMCwzYshuTAekazZ7qAi-ZAcrIcC2KqT2D1bfUX2SRzhESVi8SV0iObALFnfD8OhDXQb6llQMMOFqUNEyGCi3xDVhc2qHyxdd071MGWBpY7BJHizxjSM34msVrV_e0L0fMroXC5nUNAw8y4A=w631-h162-s-no?authuser=0)

16.確認版本
```python=
sudo apt-cache show nvidia-jetpack
```
17.安裝物件偵測程式
```python=
git clone https://github.com/dusty-nv/jetson-inference
cd jetson-inference
mkdir build
cd build
git submodule update --init
cmake ../
make -j$(nproc)
sudo make install
```
18.MIPI CAMERA功能測試
```python=
gst-launch-1.0 nvarguscamerasrc sensor_id=0 ! 'video/x-raw(memory:NVMM),width=1920, height=1080, framerate=30/1' ! nvvidconv flip-method=0 ! 'video/x-raw,width=960, height=540' ! nvvidconv ! nvegltransform ! nveglglessink -e
```
19.deepstream
```python=
sudo apt-get install libssl1.0.0  libgstreamer1.0-0 gstreamer1.0-tools gstreamer1.0-plugins-good gstreamer1.0-plugins-bad  gstreamer1.0-plugins-ugly gstreamer1.0-libav libgstrtspserver-1.0-0 libjansson4
sudo apt-get install libgstreamer1.0-0 gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav gstreamer1.0-doc gstreamer1.0-tools gstreamer1.0-x gstreamer1.0-alsa gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-qt5 gstreamer1.0-pulseaudio libgstrtspserver-1.0-dev gstreamer1.0-rtsp
sudo apt-get install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev
sudo apt-get install libgstreamer-plugins-base1.0-dev libgstreamer1.0-dev libgstrtspserver-1.0-dev libx11-dev libgstrtspserver-1.0-dev gstreamer1.0-rtsp ffmpeg
sudo apt-get install libgstreamer-plugins-base1.0-dev libgstreamer1.0-dev libgstrtspserver-1.0-dev libx11-dev libjson-glib-dev
```
20.deepstream 安裝
```python=
sudo tar -jxvf deepstream_sdk_v6.0.1_jetson.tbz2 -C /
cd /opt/nvidia/deepstream/deepstream-6.0/
sudo ./install.sh
sudo ldconfig
```
21.新增環境變數
```python=
sudo vim /etc/ld.so.conf
/opt/nvidia/deepstream/deepstream-6.0/lib/  #新增在檔案最後  
sudo ldconfig                               #立即生效

sudo vim ~/.bashrc
unset DISPLAY                               #新增在bashrc最後 
source ~/.bashrc                            #儲存退出&生效
reboot                                      #重開機
```
22.檢視deepstream版本資訊
```python=
deepstream-app --version-all
```
23.ssss
```python=
sudo apt-get install curl
. <( wget -O - https://code.headmelted.com/installers/apt.sh )
sudo apt-get install libcanberra-gtk-module 
sudo apt-get install v4l-utils
```


## 1.動機與目的
1. 動機:部門最近計畫推出應用於超商之冰箱廣告機。並汲取戴資穎所代言之喬山健身魔鏡所啟發之靈感，希望能夠針對臉部、肢體以及行為進行AI辨識，以期能夠成功應用於公司自家產品上。<br>
    ![](httsps://tinyurl.com/yc5mcue7)
2. 目的:抓取臉部框架以及特徵
    | 項目                   | 代碼          | 數值區間         |
    | ---------------------- | ------------- | ---------------- |
    | 年紀                   | age           | 0~100            |
    | 辨別碼                 | ID            | 工號或身分證字號 |
    | 性別                   | gender        | Male,Female      |
    | 口罩                   | mask          | True,False       |
    | 臉部框架 faceRactangle | 左上X座標left | 0~480            |
    |                        | 左上Y座標top  | 0~480            |
    |                        | 高height      | 0~480            |
    |                        | 寬width       | 0~480            |
    | 情緒emotion            | 開心happiness | 0~1              |
    |                        | 怒anger       | 0~1              |
    |                        | 哀sadness     | 0~1              |
    |                        | 恐懼fear      | 0~1              |
    |                        | 驚訝surprise  | 0~1              |
    
    人程式定義宣告 [Python]:
    ```python=
    class person:
        def __init__(self, age,ID ,gender,mask): 
            self.age = age
            self.ID = ID
            self.gender = gender
            self.mask = mask
    ```
    臉部框架定義宣告 [Python]:
    ```python=
    class faceRactangle:
        def __init__(self, left, top, height, weight, line_color="blue"): 
            self.left = left
            self.top = top
            self.height = height
            self.weight = weight
            self.line_color = line_color #框框框起來的顏色
    ```
    人情緒表情定義宣告 [Python]:
    ```python=
    class emotion:
        def __init__(self, happiness, anger,sadness,fear,surprise): 
            self.happiness = happiness
            self.anger = anger
            self.sadness = sadness
            self.fear = fear
            self.surprise = surprise
    ```



## 2.題目構思與方向確立
1. 大方向架構
    <br>尚待討論....
2. 硬體選擇
> Jetson Xavier NX Developer KIT[color=#f407e4]

> ![](https://tinyurl.com/yhjy9mhm)
> - 384-core NVIDIA Volta™ GPU
> - 48 Tensor Cores
> - 6-core NVIDIA ARM®v8.2 64-bit CPU 6MB L2 + 4MB L3
> - 2x NVDLA Engines
> - 7-way VLIW Vision Processor
> - 8 GB 128-bit LPDDR4x
> - 10/100/1000 Base-T Ethernet
> - 40pin GPIO: UART,I2C,I2S,SPI
> - [color=#61d8bc]


> 樹莓派800萬像素CCD[color=#f407e4]

> ![](https://tinyurl.com/yc4m95ha)
> - 800萬像素
> - MIPI CSI-2
> - 靜態圖片:3280*2464
> - 動態影片:1080p30,720p60,480p60
> - [color=#61d8bc]

3. 開發軟體工具

    | 分類       | 語言     | 使用工具                     | 使用說明       |
    | ---------- | -------- | ---------------------------- | -------------- |
    | 前端       | Python   | Anaconda                     | 深度學習       |
    | 後端排程   | Python   | Anaconda, ASP .NET Core      | 演算法優化排程 |
    | 後端資料庫 | AzureSQL | SQL Server Management Studio | 資料回饋       |
    | IOT應用    | 無       | MQTT                         | 物聯網         |


## 3.系統環境建置
2. SD卡(存取速度80MB/s)硬體切換至SSD硬碟(存取速度2800MB/s),加速執行速度,請參閱設定教學:
    >https://blog.cavedu.com/2020/06/02/jetson-xavier-nx-ssd/<br>
    >[name=CAVEDU小編][time=Mon, Jun 2, 2020]
4. 安裝Linux系統常用模組
    ```linux=
    sudo apt-get update
    sudo apt-get upgrade
    sudo apt-get install libfreetype6-dev pkg-config -y
    sudo apt-get install zlib1g-dev zip libjpeg8-dev libhdf5-dev -y
    sudo apt-get install libssl-dev libffi-dev python3-dev -y
    sudo apt-get install libhdf5-serial-dev hdf5-tools -y
    sudo apt-get install libblas-dev liblapack-dev
    sudo apt-get install libatlas-base-dev -y
    sudo apt-get install build-essential cmake libgtk-3-dev libboost-all-dev -y
    ```
4. 安裝Python必要套件模組
    ```python=
    pip install matplotlib
    pip install scikit-build
    pip install imutils
    pip install pillow
    pip install scipy
    pip install keras
    pip install scikit-learn
    pip install jupyter notebook
    pip install pycuda
    ```
5. 安裝Tensorflow
    ```python=
    pip install --extra-index-url https://developer.download.nvidia.com/compute/redist/jp/v42 tensorflow-gpu==1.13.1+nv19.4
    ```
6. todo
7. todo

## 4.資料收集與訓練學習
1. 類神經網路訓練模型:
    ```python=
    from keras.preprocessing.image import ImageDataGenerator,load_img,img_to_array
    
    def Training_Face():            #臉部訓練
        pass            
        #(img_train,label_train),(img_test,label_test)=v2.keras........
        # referense from Google
    def Training_SolarBoard():      #太陽能板熱能訓練
        pass
    ```
2. 人臉辨識:將一萬多張臉部的圖片檔進行訓練
    ```python=
    photo_path = r"Z:\face_train\face" #本機端存放人臉圖片位址
    img_format = {'png', 'jpg', 'bmp'}
    for dir_name,sub_dir_list,filename_list in os.walk(photo_path):
        if len(filename_list) > 0:
            for filename in filename_list:
                if filename.split(".")[-1] in img_format:
                    Training_Face()
    ```
3. todo
4. todo

## 5.成果展現
1. 影像輸入導入模型:
    ```python=
    import cv2,time
    import numpy as np
    video_source = "https://localhost:8080/video"
    cap, height, width = video_init(video_source)
    while (cap.isOpened()):
        status, find_face.infer = cap.read()
        if status is True:
            Detection_Face(img)#-----------進行臉部辨識
            Detection_SolarBoard(img)#-----進行太陽能板辨識
            cv2.putText(img,                         #影像
                        "我是文字",                   #文字
                        (10, 50),                    #座標
                        cv2.FONT_HERSHEY_SIMPLEX,    #字型
                        1,                           #大小
                        (0, 255, 0),                 #RGB顏色            
                        3,                           #線條寬度
                        cv2.LINE_AA)                 #線條種類
            cv2.imshow("Rafael's video", img)
        
    cap.release()
    cv2.destroyAllWindows()
    ```
2. todo
3. todo



---

<body>
<div class="alert alert-warning">
    <strong>Warning!</strong><p>There was a problem with your network connection.</p></div></body>