Use Deepstream python API to extract the model output tensor and customize model post-processing (e.g., YOLO-Pose)

# 使用Deepstream python API提取模型輸出張量並定製模型后處理（如：YOLO-Pose） ###### tags: `Edge AI` `DeepStream` `deployment` 　`Nvidia` `Jetson` `YOLO-POSE` `pose estimation` `TensorRT` `pose` ## NVIDIA Jetson 平台部署相關筆記 ### 基本環境設定 - [Jetson AGX Xavier 系統環境設定1_在windows10環境連接與安裝](https://hackmd.io/@YungHuiHsu/HJ2lcU4Rj) - [Jetson AGX Xavier 系統環境設定2_Docker安裝或從源程式碼編譯](https://hackmd.io/k-lnDTxVQDWo_V13WEnfOg) - [NVIDIA Container Toolkit 安裝筆記](https://hackmd.io/wADvyemZRDOeEduJXA9X7g) - [Jetson 邊緣裝置查詢系統性能指令jtop](https://hackmd.io/VXXV3T5GRIKi6ap8SkR-tg) - [Jetson Network Setup 網路設定](https://hackmd.io/WiqAB7pLSpm2863N2ISGXQ) - [OpenCV turns on cuda acceleration in Nvidia Jetson platform<br>OpenCV在Nvidia Jetson平台開啟cuda加速](https://hackmd.io/6IloyiWMQ_qbIpIE_c_1GA) ### 模型部署與加速 - [[Object Detection_YOLO] YOLOv7 論文筆記](https://hackmd.io/xhLeIsoSToW0jL61QRWDcQ) - [Deploy YOLOv7 on Nvidia Jetson](https://hackmd.io/kZftj6AgQmWJsbXsswIwEQ) - [Convert PyTorch model to TensorRT for 3-6x speedup<br>將PyTorch模型轉換為TensorRT，實現3-8倍加速](https://hackmd.io/_oaJhYNqTvyL_h01X1Fdmw?both) - [Accelerate multi-streaming cameras with DeepStream and deploy custom (YOLO) models<br>使用DeepStream加速多串流攝影機並部署客製(YOLO)模型](https://hackmd.io/@YungHuiHsu/rJKx-tv4h) - [Use Deepstream python API to extract the model output tensor and customize model post-processing (e.g., YOLO-Pose)<br>使用Deepstream python API提取模型輸出張量並定製模型后處理（如：YOLO-Pose）](https://hackmd.io/@YungHuiHsu/rk41ISKY2) - [Model Quantization Note 模型量化筆記](https://hackmd.io/riYLcrp1RuKHpVI22oEAXA) --- ## Github 實作 ![](https://github.com/YunghuiHsu/deepstream-yolo-pose/blob/main/imgs/Multistream_4_YOLOv8s-pose-3.PNG?raw=true =600x) ### [github/deepstream-yolo-pose](https://github.com/YunghuiHsu/deepstream-yolo-pose) - [ ] YOLO-POSE 演算法簡介(待補) - [x] 環境設定安裝請見[README](https://github.com/YunghuiHsu/deepstream-yolo-pose/blob/main/README.md) - [x] TensorRT與Deepstream教學請見: - [Convert PyTorch model to TensorRT for 3-6x speedup<br>將PyTorch模型轉換為TensorRT，實現3-8倍加速](https://hackmd.io/_oaJhYNqTvyL_h01X1Fdmw?both) - [Accelerate multi-streaming cameras with DeepStream and deploy custom (YOLO) models<br>使用DeepStream加速多串流攝影機並部署客製(YOLO)模型](https://hackmd.io/@YungHuiHsu/rJKx-tv4h) Performance on Jetson(AGX Xavier / AGX Orin) for TensorRT Engine | model | device | size | batch | fps | ms | | -------------------- |:----------:|:----:|:-----:|:-----:|:----:| | yolov8s-pose.engine | AGX Xavier | 640 | 1 | 40.6 | 24.7 | | yolov8s-pose.engine | AGX Xavier | 640 | 12 | 12.1 | 86.4 | | yolov8s-pose.engine | AGX Orin | 640 | 1 | 258.8 | 4.2 | | yolov8s-pose.engine | AGX Orin | 640 | 12 | 34.8 | 33.2 | | yolov7w-pose.pt | AGX Xavier | 640 | 1 | 14.4 | 59.8 | | yolov7w-pose.pt | AGX Xavier | 960 | 1 | 11.8 | 69.4 | | yolov7w-pose.engine* | AGX Xavier | 960 | 1 | 19.0 | 52.1 | | yolov7w-pose.engine* | AGX Orin | 960 | 1 | 61.1 | 16.8 | - \* yolov7w-pose with yolo layer tensorrt plugin from [(nanmi/yolov7-pose)](https://github.com/nanmi/yolov7-pose).NMS not included。Single batch and image_size 960 only. - test .engine(TensorRT) model with `trtexec` command.<br> - test .pt model with 15s video for baseline([test scripts](https://github.com/YunghuiHsu/yolov7/tree/log_metric)). ## 為何選擇YOLO-POSE 相比於傳統的 top-down 和 bottom-up 方法，YOLO-pose 方法具有以下優勢： 1. 計算效率更高：YOLO-pose 方法可以在單次推理中定位所有人物的位置和姿勢，而不需要多次前向傳遞。相比之下，top-down 方法需要進行多次前向傳遞，而 bottom-up 方法需要進行複雜的後處理。 2. 精度更高：YOLO-pose 方法使用了一種新的方法來處理多人姿勢估計問題，並且可以在不需要測試時間增強的情況下實現最佳性能。相比之下，top-down 和 bottom-up 方法需要使用複雜的後處理和測試時間增強技術來提高性能。 3. 複雜度更低：YOLO-pose 方法的複雜度與圖像中的人數無關，因此可以更輕鬆地處理多人姿勢估計問題。相比之下，top-down 方法的複雜度隨著圖像中的人數增加而增加，而 bottom-up 方法需要進行複雜的後處理。 **強項：多人擁擠場景** <div style="text-align: center;"> <figure> <img src="https://hackmd.io/_uploads/r1Nk0FJjh.png" alt="DS_plugin_gst-nvinfer.png" width="400"> <figcaption><span style="color: grey; ">上圖為yolov5m6Pose、下圖為HigherHRNetW32，可以見到在多人擁擠場景的情況下，傳統的由下而上(Bottom-Up)方法，在關節點分群時錯誤率較高</span></figcaption> </figure> </div> <div style="text-align: center;"> <figure> <img src="https://hackmd.io/_uploads/HkHFpFkj2.png" alt="DS_plugin_gst-nvinfer.png" width="800"> <figcaption><span style="color: BLACK; font-weight: bold;">YOLO-POSE架構</span></figcaption> </figure> </div> ## Depstream Python API for Extracting Model Output Tensor 完整程式碼見 : [deepstream_YOLOv8-Pose_rtsp.py](https://github.com/YunghuiHsu/deepstream-yolo-pose/blob/main/deepstream_YOLOv8-Pose_rtsp.py#L92) ### Extracting Model Output Tensor 要從deepstream buffer中讀取模型預測結果，需要在Deepstream Pipeline中負責模型推理的[`Gst-nvinfer`](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvinfer.html)模塊輸出的Buffer中將meta data撈出 <div style="text-align: center;"> <figure> <img src="https://docs.nvidia.com/metropolis/deepstream/dev-guide/_images/DS_plugin_gst-nvinfer.png" alt="DS_plugin_gst-nvinfer.png" width="450"> <figcaption><span style="color: green; font-weight: bold;">Gst-nvinfer</span></figcaption> </figure> </div> 整體data pipeline如下圖。如果還想連影像資料一起撈的話，要從FRAME BUFFER著手 <div style="text-align: center;"> <figure> <img src="https://github.com/NVIDIA-AI-IOT/deepstream_python_apps/blob/master/apps/deepstream-imagedata-multistream/imagedata-app-block-diagram.png?raw=true" alt="imagedata-app-block-diagram.png" width="800"> <figcaption><span style="color: grey; font-weight: bold;"> Workflow for Extracting Model Output Tensor from `Gst-nvinfer`</span></figcaption> </figure> </div> source : [deepstream-imagedata-multistream](https://github.com/NVIDIA-AI-IOT/deepstream_python_apps/tree/master/apps/deepstream-imagedata-multistream) #### 1. Parsing the output data of a custom model from the `NvDsBatchMeta` data structure<br>從`NvDsBatchMeta`資料結構解析取出客製模型的輸出資料要取得的目標(模型預測的張量)位於 `NvDsBatchMeta` > `NvDsFrameMeta` > `NvDsUserMeta` 之下，並指定為`NvDsInferTensorMeta`的資料結構，該結構用於存儲推論張量的相關屬性和數據。 ##### `NvDsBatchMeta` Deepstream中關於Meta data資料結構定義如下圖 <div style="text-align: center;"> <figure> <img src="https://docs.nvidia.com/metropolis/deepstream/dev-guide/_images/DS_plugin_metadata.png" alt="DS_plugin_metadata.png" width="600"> <figcaption><span style="color: grey; font-weight: bold;"> NvDsBatchMeta: Basic Metadata Structure</span></figcaption> </figure> </div> source : [MetaData in the DeepStream SDK](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_metadata.html) ##### `NvDsUserMeta` - DeepStream SDK API Reference關於`NvDsUserMeta`的資料結構定義 <div style="text-align: center;"> <figure> <img src="https://docs.nvidia.com/metropolis/deepstream/dev-guide/sdk-api/struct__NvDsUserMeta__coll__graph.png" alt="struct__NvDsUserMeta__coll__graph.png" width="400"> <figcaption><span style="color: grey; font-weight: bold;"> NvDsBatchMeta: Basic Metadata Structure</span></figcaption> </figure> </div> source : [NVIDIA DeepStream SDK API Reference - NvDsUserMeta Struct Reference](https://docs.nvidia.com/metropolis/deepstream/dev-guide/sdk-api/struct__NvDsUserMeta.html) ###### 自定義`NvDsUserMeta`範例 :::spoiler - Nvidia bodypose2D 自定義meta資料結構 ```cpp static gpointer nvds_copy_2dpose_meta (gpointer data, gpointer user_data) { NvDsUserMeta *user_meta = (NvDsUserMeta *) data; NvDs2DposeMetaData *p_2dpose_meta_data = (NvDs2DposeMetaData *)user_meta->user_meta_data; NvDs2DposeMetaData *pnew_2dpose_meta_data = (NvDs2DposeMetaData *)g_memdup ( p_2dpose_meta_data, sizeof(NvDs2DposeMetaData)); return (gpointer) pnew_2dpose_meta_data; } ``` - 使用obj\_user\_meta\_list取出meta資料 ```cpp for (l_user = obj_meta->obj_user_meta_list; l_user != NULL; l_user = l_user->next) ``` ::: * 從\`NvDsFrameMeta\` 取出Tensor Metadata data的python API， ###### [To read or parse inference raw tensor data of output layers](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvinfer.html) 詳細的範例見DeepStream SDK samples`sources/apps/sample_apps/deepstream_infer_tensor_meta-test.cpp` 以下是讀取及解析模型預測結果的raw tensor data方法 - 為`Gst-nvinfer`插件啟用屬性`output-tensor-meta`，或在配置文件中啟用相同名稱的屬性 ``` output-tensor-meta=1 ``` - 取出方式 - 當作為**主要GIE**時 - 每幀`NvDsFrameMeta`內的`frame_user_meta_list` - 當作為**次要GIE**時 - 每幀`NvDsObjectMeta`內的`obj_user_meta_list` - `Gst-nvinfer`的Meta數據可以在下游的`GStreamer pad probe`中訪問 \* GIE: GPU Inference Engine :::spoiler 範例: bodypose2d中直接撈出模型原始輸出的張量做操作 - [deepstream_bodypose2d_app.cpp](https://github.com/NVIDIA-AI-IOT/deepstream_tao_apps/blob/d6094a4f9e52788ec9c42052042f9252022d04d8/apps/tao_others/deepstream-bodypose2d-app/deepstream_bodypose2d_app.cpp#L298) ```cpp=298 /* Iterate each frame metadata in batch */ for (NvDsMetaList *l_frame = batch_meta->frame_meta_list; l_frame != NULL; l_frame = l_frame->next) { NvDsFrameMeta *frame_meta = (NvDsFrameMeta *)l_frame->data; . . . /* Iterate user metadata in frames to search PGIE's tensor metadata */ for (NvDsMetaList *l_user = frame_meta->frame_user_meta_list; l_user != NULL; l_user = l_user->next) { NvDsUserMeta *user_meta = (NvDsUserMeta *)l_user->data; if (user_meta->base_meta.meta_type != NVDSINFER_TENSOR_OUTPUT_META) continue; NvDsInferTensorMeta *meta = (NvDsInferTensorMeta *)user_meta->user_meta_data; . . . for (unsigned int i = 0; i < meta->num_output_layers; i++) { NvDsInferLayerInfo *info = &meta->output_layers_info[i]; info->buffer = meta->out_buf_ptrs_host[i]; NvDsInferDimsCHW LayerDims; std::vector<NvDsInferLayerInfo> ``` ::: - python code example for extract `NvDsInferTensorMeta` ```python def pose_src_pad_buffer_probe(pad, info, u_data): gst_buffer = info.get_buffer() batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer)) l_frame = batch_meta.frame_meta_list frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data) l_usr = frame_meta.frame_user_meta_list user_meta = pyds.NvDsUserMeta.cast(l_usr.data) tensor_meta = pyds.NvDsInferTensorMeta.cast(user_meta.user_meta_data) ``` - ps: ["nvdsparsebbox_Yolo.cpp"](https://github.com/NVIDIA-AI-IOT/yolo_deepstream/blob/5af35bab7f6dfca7f1f32d44847b2a91786485f4/deepstream_yolo/nvdsinfer_custom_impl_Yolo/nvdsparsebbox_Yolo.cpp#L227)中是在`[Gst-nvinfer`\]後的\[`Gst-nvdspostprocess`\]階段，取出`NvDsInferTensorMeta` 內的`output_layers_info`進行解析 #### 2. Convert `NvDsBatchMeta` data schema to `numpy` array<br>將`NvDsBatchMeta`資料結構解轉換為`numpy` 陣列 `NvDsInferTensorMeta` > `NvDsInferLayerinfo` > `layer.buffer` > `numpy array` ##### `NvDsInferTensorMeta` <div style="text-align: center;"> <figure> <img src="https://docs.nvidia.com/metropolis/deepstream/dev-guide/sdk-api/structNvDsInferTensorMeta__coll__graph.png" alt="struct__NvDsUserMeta__coll__graph.png" width="400"> <figcaption><span style="color: grey; font-weight: bold;"> NvDsInferTensorMeta </span></figcaption> </figure> </div> source : [NVIDIA DeepStream SDK API Reference - NvDsInferTensorMeta Struct Reference](https://docs.nvidia.com/metropolis/deepstream/dev-guide/sdk-api/structNvDsInferTensorMeta.html) ##### 將模型的預測結果(位於**l**ayer.buffer記憶體區塊)轉變為numpy陣列在取得模型推理結果的張量後(Deepstream中用來儲存推理結果的`NvDsInferTensorMeta`資料結構)，這時候的原始資料型態還是記憶體中分配的一塊buffer，必須透過python `ctype` 函式庫的幫助，取得buffer記憶體位置的指標後，將其轉換為numpy資料格式，方便後續在python中處理 ##### `NvDsInferTensorMeta` > numpy array : python code example :::spoiler - **pyds.get_ptr(layer.buffer)**：這個函式是用於獲取指向**layer.buffer**的指標，**layer.buffer**表示一塊記憶體區域。 - **ctypes.cast(pointer, POINTER(ctypes.c\_float))**：**ctypes.cast**函式用於將指標進行類型轉換。在這個例子中，我們將獲取的指標**pyds.get\_ptr(layer.buffer)轉換為指向ctypes.c\_float**類型的指標。這樣做是為了讓後續的處理能夠以**float**類型進行。 - **np.ctypeslib.as\_array(ptr, shape=dims)**：這個函式將指標**ptr**轉換為NumPy陣列。**dims** 可利用`layer_output_info.inferDims.d`API來自動抓取模型輸出的形狀，用於指定NumPy陣列的維度。這樣，後面就可以使用`NumPy`提供的強大功能來操作這個陣列/張量，即模型預測結果) ```python! data_type_map = {pyds.NvDsInferDataType.FLOAT: ctypes.c_float, pyds.NvDsInferDataType.INT8: ctypes.c_int8, pyds.NvDsInferDataType.INT32: ctypes.c_int32} def pose_src_pad_buffer_probe(pad, info, u_data): . . . tensor_meta = pyds.NvDsInferTensorMeta.cast(user_meta.user_meta_data) # layers_info = [] # for i in range(tensor_meta.num_output_layers): # layer = pyds.get_nvds_LayerInfo(tensor_meta, i) # # print(i, layer.layerName) # layers_info.append(layer) # if your model only have one output layer, just give "0" as key layer_output_info = pyds.get_nvds_LayerInfo(tensor_meta, 0) # as num_output_layers == 1 # remove zeros from both ends of the array. 'b' : 'both' dims = np.trim_zeros(layer_output_info.inferDims.d, 'b') if frame_number == 0 : print(f'\tModel output dimension from LayerInfo: {dims}') # load float* buffer to python cdata_type = data_type_map[layer_output_info.dataType] ptr = ctypes.cast(pyds.get_ptr(layer_output_info.buffer), ctypes.POINTER(cdata_type)) # Determine the size of the array # Automatic acquisition of buffer memory sizes based on output in the model out = np.ctypeslib.as_array(ptr, shape=dims) ``` ::: #### 3. Customized Tensor Post-Processing based on Model Output <br> 根據模型輸出張量的客置後處理檢視[YOLOv8-pose](https://github.com/ultralytics/ultralytics)的輸出，張量形狀為(batch, anchors, max_outputs) ![](https://hackmd.io/_uploads/Bym3Mynt3.png =800x) `netron yolov8s-pose.onnx` ![](https://hackmd.io/_uploads/ryDZ712Yn.png =400x) 由於`yolov7-pose`與`yolov8-pose`模型的輸出形狀與anchors不同我這邊使用的後處理函式需要採用(batch, max_outputs, anchors)的格式輸入，而且包含類別標籤的機率，因此手動客製調整`yolov8-pose`模型的輸出張量 :::info - 人體關節鍵點keypoints YOLO-POSE的人體關節點17個，每個點位包含{$x,y,conf$}，其中conf為0、1或2，分別代表"不存在"、"存在但被遮擋"或"存在且不被遮擋"，故和人體有關的label一個人包含17x3（3代表x、y、conf）=51個元素，bbox預測的label含有6個元素(x,y,w,h,box_conf,cls_conf)，共51+6=57個元素，表達如下： $$ P_{v} = \{C_{x}, C_{y}, W, H, box_{conf}, class_{conf}, K_{x}^{1}, K_{y}^{1}, K_{conf}^{1}, \cdots, K_{x}^{n}, K_{y}^{n}, K_{conf}^{n}\} $$ ![](https://hackmd.io/_uploads/B1aRBJntn.png =400x) ![](https://hackmd.io/_uploads/SJgRLT3Y2.png =400x) - anchors of yolov7-pose - 57 # bbox(4) + confidence(1) + cls(1) + keypoints(3 x 17) = 4 + 1 + 1 + 51 = 57 - anchors of yolov8-pose - 56 # bbox(4) + confidence(1) + keypoints(3 x 17) = 4 + 1 + 0 + 51 = 56 ::: ```python= def pose_src_pad_buffer_probe(pad, info, u_data): . network_info = tensor_meta.network_info . # [Optional] Postprocess for yolov8-pose prediction tensor # [YOLOv8](https://github.com/ultralytics/ultralytics) # 　(batch, 56, 8400)　＞(batch, 8400, 56) for yolov8 out = out.transpose((0, 2, 1)) # make pseudo class prob cls_prob = np.ones((out.shape[0], out.shape[1], 1), dtype=np.uint8) out[..., :4] = map_to_zero_one(out[..., :4]) # scalar prob to [0, 1] # insert pseudo class prob into predictions out = np.concatenate((out[..., :5], cls_prob, out[..., 5:]), axis=-1) out[..., [0, 2]] = out[..., [0, 2]] * network_info.width # scale to screen width out[..., [1, 3]] = out[..., [1, 3]] * network_info.height # scale to screen height ``` ## config file [dstest1_pgie_YOLOv8-Pose_config.txt](https://github.com/YunghuiHsu/deepstream-yolo-pose/blob/main/configs/dstest1_pgie_YOLOv8-Pose_config.txt) 以下列出幾個使用客製模型時需要特別設置的參數 #### `model-engine-file` 指定你的TensorRT Engine路徑，我習慣放在`/opt/nvidia/deepstream/deepstream/samples/models/`並指定絕對路徑 - `model-engine-file=<your engine path>` #### `network-type` 指定模型種類，由於deepstream還沒有提供公版的姿態模型，所有要指定其他:`100` > [Gst-nvdspreprocess & Gst-nvdspostprocess plugin] > 0=Detector, 1=Classifier, 2=Segmentation, 100=Other - `network-type=100` #### `output-tensor-meta` 要設為`1`(true)才能在`NvDsUserMeta`取得模型輸出結果 - `output-tensor-meta=1` #### `tensor-meta-pool-size` 預設單一batch是6，即使不設定也會自動分配，有測試過設比較大的值但加速效果不明顯 > if batch-size!=1 。 ## Reference #### [deepstream-imagedata-multistream](https://github.com/NVIDIA-AI-IOT/deepstream_python_apps/tree/master/apps/deepstream-imagedata-multistream) - 取出影像資料後處理的範例 #### [deepstream-ssd-parser](https://github.com/NVIDIA-AI-IOT/deepstream_python_apps/tree/master/apps/deepstream-ssd-parser) - 內有自定義解析Metadata的方式可以參考 #### [NVIDIA DeepStream SDK API Reference ](https://docs.nvidia.com/metropolis/deepstream/dev-guide/sdk-api/index.html) #### [DEEPSTREAM PYTHON API REFERENCE/NvDsInfer](https://docs.nvidia.com/metropolis/deepstream/python-api/PYTHON_API/NvDsInfer/NvDsInfer_toc.html) #### [Using a Custom Model with DeepStream](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_using_custom_model.html)

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.