---
# System prepended metadata

title: Greengrass + Openvino on Ubuntu
tags: [Tutorial]

---

###### tags: `Tutorial`

# Greengrass + Openvino on Ubuntu

## 全新安裝

參考此說明修改 : https://software.intel.com/en-us/articles/get-started-with-the-openvino-toolkit-and-aws-greengrass

1. 安裝openvino(參考intel網站)
2. 轉換IR
3. 將IR壓縮成zip上傳到S3上
4. 設定AWS Greengrass
5. 啟動AWS Greengrass
6. 將位於
`/opt/intel/computer_vision_sdk_2018.5.445/deployment_tools/inference_engine/samples/python_samples/greengrass_samples` 下的 `greengrass_object_detection_sample_ssd.py`與greengrasssdk合併打包成zip檔上傳到Lambda上
7. 修改`greengrass_object_detection_sample_ssd.py`檔案 : 
    
    *   將  
        ```python
        n, c, h, w = net.inputs[input_blob]
        ```
        修改為
        ```python
        n, c, h, w = net.inputs[input_blob].shape
        ```
    *   新增以下程式碼位於`frame_count += 1`之後
        ```python
        cv2.imshow('Greengrass_Openvino', frame)
        if cv2.waitKey(1) & 0xFF == ord('q'):
            break
        ```
        ![](https://i.imgur.com/ZObKB6D.png)
        
    *    新增文字訊息
         將
         ```python
         cv2.putText(frame, ("label:"+ str(int(obj[1]))+", class:"+ str(classlabel)+ ", confidence"+ str(round(obj[2], 2))), (xmin, ymin - 10), cv2.FONT_HERSHEY_DUPLEX, 0.6, (255, 255, 255), 2, cv2.LINE_AA)
         cv2.putText(frame, ("label:"+ str(int(obj[1]))+", class:"+ str(classlabel)+ ", confidence"+ str(round(obj[2], 2))), (xmin, ymin - 10), cv2.FONT_HERSHEY_DUPLEX, 0.6, (0, 72, 97), 1, cv2.LINE_AA)
         ```
         插入在`object_id += 1` 之後
         然後將
         ```python
         cv2.putText(frame, ("inference_fps:"+ str(inference_fps)), (int(20), int(20)), cv2.FONT_HERSHEY_DUPLEX, 0.6, (255, 255, 255), 2, cv2.LINE_AA)
         cv2.putText(frame, ("inference_fps:"+ str(inference_fps)), (int(20), int(20)), cv2.FONT_HERSHEY_DUPLEX, 0.6, (1, 66, 0), 1, cv2.LINE_AA)
         ```
         插入在`frame_count += 1` 與 `cv2.imshow('Greengrass_Openvino', frame)` 之間
         然後將
         ```python
         inference_fps = 0
         ```
         插入在`while (cap.isOpened()):` 之前
         然後將
         ```python
         inference_fps = res_json["inference_fps"]
         ```
         插入在`res_json["inference_fps"] = frame_count / inf_seconds` 之後
    *    完整Lambda Function : 
            ```python
            """
            BSD 3-clause "New" or "Revised" license

            Copyright (C) 2018 Intel Coporation.

            Redistribution and use in source and binary forms, with or without
            modification, are permitted provided that the following conditions are met:

            * Redistributions of source code must retain the above copyright notice, this
              list of conditions and the following disclaimer.

            * Redistributions in binary form must reproduce the above copyright notice,
              this list of conditions and the following disclaimer in the documentation
              and/or other materials provided with the distribution.

            * Neither the name of the copyright holder nor the names of its
              contributors may be used to endorse or promote products derived from
              this software without specific prior written permission.

            THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
            AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
            IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
            DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
            FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
            DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
            SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
            CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
            OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
            OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
            """

            import sys
            import os
            import cv2
            import numpy as np
            import greengrasssdk
            import boto3
            import timeit
            import datetime
            import json
            from collections import OrderedDict 

            from openvino.inference_engine import IENetwork, IEPlugin

            # Specify the delta in seconds between each report
            reporting_interval = 1.0

            # Parameters for IoT Cloud
            enable_iot_cloud_output = True

            # Parameters for Kinesis
            enable_kinesis_output = False
            kinesis_stream_name = ""
            kinesis_partition_key = ""
            kinesis_region = ""

            # Parameters for S3
            enable_s3_jpeg_output = False
            s3_bucket_name = "ssd_test"

            # Parameters for jpeg output on local disk
            enable_local_jpeg_output = False

            # Create a Greengrass Core SDK client for publishing messages to AWS Cloud
            client = greengrasssdk.client("iot-data")

            # Create an S3 client for uploading files to S3
            if enable_s3_jpeg_output:
                s3_client = boto3.client("s3")

            # Create a Kinesis client for putting records to streams
            if enable_kinesis_output:
                kinesis_client = boto3.client("kinesis", "us-west-2")

            # Read environment variables set by Lambda function configuration
            PARAM_MODEL_XML = os.environ.get("PARAM_MODEL_XML")
            PARAM_INPUT_SOURCE = os.environ.get("PARAM_INPUT_SOURCE")
            PARAM_DEVICE = os.environ.get("PARAM_DEVICE")
            PARAM_OUTPUT_DIRECTORY = os.environ.get("PARAM_OUTPUT_DIRECTORY")
            PARAM_CPU_EXTENSION_PATH = os.environ.get("PARAM_CPU_EXTENSION_PATH")
            PARAM_LABELMAP_FILE = os.environ.get("PARAM_LABELMAP_FILE")
            PARAM_TOPIC_NAME = os.environ.get("PARAM_TOPIC_NAME", "intel/faas/ssd")

            def report(res_json, frame):
                now = datetime.datetime.now()
                date_prefix = str(now).replace(" ", "_")
                if enable_iot_cloud_output:
                    data = json.dumps(res_json)
                    client.publish(topic=PARAM_TOPIC_NAME, payload=data)
                if enable_kinesis_output:
                    kinesis_client.put_record(StreamName=kinesis_stream_name, Data=json.dumps(res_json), PartitionKey=kinesis_partition_key)
                if enable_s3_jpeg_output:
                    temp_image = os.path.join(PARAM_OUTPUT_DIRECTORY, "inference_result.jpeg")
                    cv2.imwrite(temp_image, frame)
                    with open(temp_image) as file:
                        image_contents = file.read()
                        s3_client.put_object(Body=image_contents, Bucket=s3_bucket_name, Key=date_prefix + ".jpeg") 
                if enable_local_jpeg_output:
                    cv2.imwrite(os.path.join(PARAM_OUTPUT_DIRECTORY, date_prefix + ".jpeg"), frame)


            def greengrass_object_detection_sample_ssd_run():
                client.publish(topic=PARAM_TOPIC_NAME, payload="OpenVINO: Initializing...")
                model_bin = os.path.splitext(PARAM_MODEL_XML)[0] + ".bin"

                # Plugin initialization for specified device and load extensions library if specified
                plugin = IEPlugin(device=PARAM_DEVICE, plugin_dirs="")
                if "CPU" in PARAM_DEVICE:
                    plugin.add_cpu_extension(PARAM_CPU_EXTENSION_PATH)
                # Read IR
                net = IENetwork(model=PARAM_MODEL_XML, weights=model_bin)
                assert len(net.inputs.keys()) == 1, "Sample supports only single input topologies"
                assert len(net.outputs) == 1, "Sample supports only single output topologies"
                input_blob = next(iter(net.inputs))
                out_blob = next(iter(net.outputs))
                # Read and pre-process input image
                n, c, h, w = net.inputs[input_blob].shape
                cap = cv2.VideoCapture(PARAM_INPUT_SOURCE)
                exec_net = plugin.load(network=net)
                del net
                client.publish(topic=PARAM_TOPIC_NAME, payload="Starting inference on %s" % PARAM_INPUT_SOURCE)
                start_time = timeit.default_timer()
                inf_seconds = 0.0
                frame_count = 0
                labeldata = None
                if PARAM_LABELMAP_FILE is not None:
                   with open(PARAM_LABELMAP_FILE) as labelmap_file:
                        labeldata = json.load(labelmap_file)

                inference_fps = 0

                while (cap.isOpened()):
                    ret, frame = cap.read()
                    if not ret:
                        break
                    frameid = cap.get(cv2.CAP_PROP_POS_FRAMES)
                    initial_w = cap.get(3)
                    initial_h = cap.get(4)
                    in_frame = cv2.resize(frame, (w, h))
                    in_frame = in_frame.transpose((2, 0, 1))  # Change data layout from HWC to CHW
                    in_frame = in_frame.reshape((n, c, h, w))
                    # Start synchronous inference
                    inf_start_time = timeit.default_timer()
                    res = exec_net.infer(inputs={input_blob: in_frame})
                    inf_seconds += timeit.default_timer() - inf_start_time
                    # Parse detection results of the current request
                    res_json = OrderedDict() 
                    frame_timestamp = datetime.datetime.now()    
                    object_id = 0
                    for obj in res[out_blob][0][0]:
                         if obj[2] > 0.5:
                             xmin = int(obj[3] * initial_w)
                             ymin = int(obj[4] * initial_h)
                             xmax = int(obj[5] * initial_w)
                             ymax = int(obj[6] * initial_h)
                             cv2.rectangle(frame, (xmin, ymin), (xmax, ymax), (255, 165, 20), 4)
                             obj_id = "Object" + str(object_id)
                             classlabel = labeldata[str(int(obj[1]))] if labeldata else ""
                             res_json[obj_id] = {"label": int(obj[1]), "class": classlabel, "confidence": round(obj[2], 2), "xmin": round(obj[3], 2), "ymin": round(obj[4], 2), "xmax": round(obj[5], 2), "ymax": round(obj[6], 2)}
                             object_id += 1
                             cv2.putText(frame, ("label:"+ str(int(obj[1]))+", class:"+ str(classlabel)+ ", confidence"+ str(round(obj[2], 2))), (xmin, ymin - 10), cv2.FONT_HERSHEY_DUPLEX, 0.6, (255, 255, 255), 2, cv2.LINE_AA)
                             cv2.putText(frame, ("label:"+ str(int(obj[1]))+", class:"+ str(classlabel)+ ", confidence"+ str(round(obj[2], 2))), (xmin, ymin - 10), cv2.FONT_HERSHEY_DUPLEX, 0.6, (0, 72, 97), 1, cv2.LINE_AA)
                    frame_count += 1
                    cv2.putText(frame, ("inference_fps:"+ str(inference_fps)), (int(20), int(20)), cv2.FONT_HERSHEY_DUPLEX, 0.6, (255, 255, 255), 2, cv2.LINE_AA)
                    cv2.putText(frame, ("inference_fps:"+ str(inference_fps)), (int(20), int(20)), cv2.FONT_HERSHEY_DUPLEX, 0.6, (1, 66, 0), 1, cv2.LINE_AA)
                    cv2.imshow('Greengrass_Openvino', frame)
                    if cv2.waitKey(1) & 0xFF == ord('q'):
                        break

                    # Measure elapsed seconds since the last report
                    seconds_elapsed = timeit.default_timer() - start_time
                    if seconds_elapsed >= reporting_interval:
                        res_json["timestamp"] = frame_timestamp.isoformat()
                        res_json["frame_id"] = int(frameid)   
                        res_json["inference_fps"] = frame_count / inf_seconds
                        inference_fps = res_json["inference_fps"]
                        start_time = timeit.default_timer()
                        report(res_json, frame)
                        frame_count = 0
                        inf_seconds = 0.0

                client.publish(topic=PARAM_TOPIC_NAME, payload="End of the input, exiting...")
                del exec_net
                del plugin

            greengrass_object_detection_sample_ssd_run()

            def function_handler(event, context):
                client.publish(topic=PARAM_TOPIC_NAME, payload='HANDLER_CALLED!')
                return

            ```

8. 參照AWS GG使用手冊發布並與Greengress 群組連接
9. 設定AWS GG Lambda:

    *    執行身分與Lambda 生命週期:
         ![](https://i.imgur.com/bvTPE5U.png)
    *    環境變數
            *   | Key | Value | 
                | -------- | -------- |
                | LD_LIBRARY_PATH     | /opt/intel/computer_vision_sdk/opencv/share/OpenCV/3rdparty/lib:/opt/intel/computer_vision_sdk/opencv/lib:/opt/intel/opencl:/opt/intel/computer_vision_sdk/deployment_tools/inference_engine/external/cldnn/lib:/opt/intel/computer_vision_sdk/deployment_tools/inference_engine/external/mkltiny_lnx/lib:/opt/intel/computer_vision_sdk/deployment_tools/inference_engine/lib/ubuntu_16.04/intel64:/opt/intel/computer_vision_sdk/deployment_tools/model_optimizer/model_optimizer_caffe/bin:/opt/intel/computer_vision_sdk/openvx/lib     |
                | PYTHONPATH     | /opt/intel/computer_vision_sdk/python/python2.7/ubuntu16/     |
                | PARAM_CPU_EXTENSION_PATH     | /opt/intel/computer_vision_sdk/deployment_tools/inference_engine/lib/ubuntu_16.04/intel64/libcpu_extension_avx2.so     |
                | PARAM_MODEL_XML     | /aimodel/SqueezeNetSSD-5Class.xml     |
                | PARAM_INPUT_SOURCE     | /dev/video0     |
                | PARAM_DEVICE     | CPU     |
                | PARAM_OUTPUT_DIRECTORY     | /home/iei/dl/img     |
                | PARAM_NUM_TOP_RESULTS     | 3     |
                | PARAM_TOPIC_NAME     | intel/faas/ssd     |
                | DISPLAY     | unix:0     |
                | XAUTHORITY     | /X11A/.Xauthority     |
                
    *    設定資源
            *   Name | Resource Type | Local Path | Access
                ------------- |------------- | ------------- | -------------
                Webcam | Device | `/dev/video0` | Read and Write
                DataDir | Volume | `/opt/intel/computer_vision_sdk/` | Read and Write
                OpenVINOPath | Volume | `/opt/intel/computer_vision_sdk/` | Read-Only
                GPU | Device | `/dev/dri/renderD128` | Read and Write
                X11 | Volume | `/tmp/.X11-unix/` | Read and Write
                X11A | Volume | Source : `/home/iei/.Xauthority` Destination : `/X11A/.Xauthority` | Read and Write
                
    *    設定 Machine Learning Resour
            *    選擇上傳的IR .zip檔案
            *    Path設定為 : `/aimodel/`
            *    與[Lambda Function]綁定
            *    Read-Only

10. 設定Subscribe
    * **Source** : [Lambda Function]
    * **Destination** : IoT Cloud
    * **Topic** : intel/faas/ssd

11. Depory

    ![](https://i.imgur.com/IrRIwVm.png)
    
12. 開機自啟動，參閱 : https://docs.aws.amazon.com/zh_tw/greengrass/latest/developerguide/gg-core.html#start-on-boot


## 已安裝完啟動步驟

### Host 步驟

* Username/Password : iei/123
* 正常開機會自啟動
* 若沒自啟動輸入以下command :
    ```bash
    sudo systemctl restart greengrass.service
    ```
* 查閱啟動狀況輸入以下command :
    ```bash
    sudo systemctl status greengrass.service
    ```
* 上述命令啟動失敗輸入以下命令手動啟動並查看啟動狀況 : 
    ```bash
    sudo /greengrass/ggc/core/greengrassd start
    ```
* 關閉Greengrass Core : 
    ```bash
    sudo systemctl stop greengrass.service
    ```

### AWS IoT 步驟

* 進入IoT Greengrass頁面
![](https://i.imgur.com/LB1FnNq.png)

* 切換到Test頁面，Subscribe `intel/faas/ssd`
![](https://i.imgur.com/SaAk6sa.png)

* 若成功啟動即會看到資訊
![](https://i.imgur.com/iX5QwdN.png)
