--- # System prepended metadata title: OpenPoseCMU tags: [CS] --- --- title: 'OpenPoseCMU' tags: CS --- # Table of Contents [TOC] ## Referencd [github](https://github.com/CMU-Perceptual-Computing-Lab/openpose) [demo](https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/demo_overview.md) [openpose documentation(ENG)](https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/python/openpose/openpose_python.cpp#L194) [openpose documentation(CHI)](https://blog.csdn.net/weixin_40802676/article/details/100830688) ## 人體姿態辨識論文 [Hung-Chih Chiu](https://medium.com/@williamchiu0127) ## LightWeight OpenPose [Real-time 2D Multi-Person Pose Estimation on CPU: Lightweight OpenPose](https://arxiv.org/pdf/1811.12004.pdf) ## Openpose Windows 安裝 ### Some problems 1. 要安裝python API時顯示找不到pyopenpose ![](https://i.imgur.com/gFnDQqq.png) 打開visual studio，在pyopenpose這個專案進行build ![](https://i.imgur.com/i5ETKt2.png) 資料夾會出現pyopenpose的library ![](https://i.imgur.com/p84q1tc.png) ### 添加環境變數 1. 把openpose.dll所在資料夾加到使用者變數Path裡面 ![](https://i.imgur.com/7FN5RSa.png) 2. 把以下三個檔案所再資料夾加到系統變數PYTHONPATH裡面 ![](https://i.imgur.com/vORmIbK.png) ![](https://i.imgur.com/ZazerEx.png) 添加完畢之後就可以直接import openpose了 ```python import pyopenpose as op ``` ## Openpose save video in different format - The default video save format is `.avi`. If you want to save it with `.mp4`, error will show as below: ![](https://i.imgur.com/Sz0OYXw.png) - Solution: Install `ffmpeg` > sudo apt-get install ffmpeg ## Openpose 不同model [model link](https://github.com/CMU-Perceptual-Computing-Lab/openpose_train/tree/master/experimental_models) ![](https://i.imgur.com/At0eplV.png) 1. 100_135AlmostSameBatchAllGPUs - [Paper link(whole body in 2019)](https://arxiv.org/pdf/1909.13423.pdf) - Feature extraction : 10 VGG layers - 4 wider & deeper PAF stages and 1 CM stage 3. 1_25BBkg - [Paper link(whole body in 2019)](https://arxiv.org/pdf/1909.13423.pdf) - Feature extraction : 10 VGG layers 4. 1_25BSuperModel11FullVGG - [Paper link(whole body in 2019)](https://arxiv.org/pdf/1909.13423.pdf) - Feature extraction : Complete VGG layers 5. body_25() - [Paper link(Pami那篇)](https://arxiv.org/pdf/1812.08008.pdf) - Feature extraction : 10 VGG layers ## Openpose src code [prototext](https://github.com/CMU-Perceptual-Computing-Lab/openpose_caffe_train/blob/master/src/caffe/proto/caffe.proto) [oPDataTransformer](https://github.com/CMU-Perceptual-Computing-Lab/openpose_caffe_train/blob/master/src/caffe/openpose/oPDataTransformer.cpp) [dataAugmentation](https://github.com/CMU-Perceptual-Computing-Lab/openpose_caffe_train/blob/master/src/caffe/openpose/dataAugmentation.cpp) ## Openpose Detection Parts ### COCO (Original)(17) ![](https://i.imgur.com/wi7vQmL.png) ### COCO used in Openpose(18) ![](https://i.imgur.com/r4zHc8w.png) ### BODY25(25:COCO + middle of hips + 6 foot parts) ![](https://i.imgur.com/w4LTKiO.png) ## Openpose Linux (Ubuntu) 安裝遇到的小問題 1. generate完成的時候發現跑src code，圖片會顯示不出來 + Sol:原因是opencv的版本太新(4.3.0)，降回4.2.0就可以正常顯示了 2. generate完成之後，跑src code，會發現只顯示原本的圖片，上面並沒有畫上skeleton + Sol:因為前面在下載model的時候有失敗(coco和mpi)，只要重新下載並放到正確的位置(models/......)就可以得到正常的結果了 ## Openpose Training - Provided Dataset [github](https://github.com/CMU-Perceptual-Computing-Lab/openpose_train) ### [Training Process](https://github.com/CMU-Perceptual-Computing-Lab/openpose_train/blob/master/training/README.md) 1. Get images 2. Annotate 3. Generate LMDB files 4. GPUing... ### Prepare Openpose Training File [openpose_train.md](https://github.com/CMU-Perceptual-Computing-Lab/openpose_train/blob/master/training/README.md) 1. Generate LMDB files using COCO dataset ([REF](https://www.immersivelimit.com/tutorials/create-coco-annotations-from-scratch/#coco-dataset-format)) ![](https://i.imgur.com/jMo8C86.png) - In step b, to find images without people. - In step e, to obtain json file for training. - For original COCO annotation (x, y, v), - v=0: not labeled (in which case x=y=0), - v=1: labeled but not visible - v=2: labeled and visible - For refined json file, - v=0: labeled but not visible - v=1: labeled and visible - v=2: not labeled (in which case x=y=0), - ex. 000000345507.jpg ![](https://i.imgur.com/r8zFKyJ.png) - keypoints - "keypoints": [ "nose","left_eye","right_eye","left_ear","right_ear", "left_shoulder","right_shoulder","left_elbow","right_elbow", "left_wrist","right_wrist","left_hip","right_hip", "left_knee","right_knee","left_ankle","right_ankle" ] - original - [115.000,170.000,2.000], [0.000,0.000,0.000], [107.000,161.000,2.000], [0.000,0.000,0.000], [74.000,165.000,2.000], [63.000,225.000,2.000], [76.000,240.000,2.000], [81.000,325.000,2.000], [94.000,342.000,2.000], [109.000,390.000,1.000], [140.000,413.000,2.000], [80.000,391.000,1.000], [92.000,416.000,2.000], [0.000,0.000,0.000], [0.000,0.000,0.000], [0.000,0.000,0.000], [0.000,0.000,0.000] - refined - [115.000,170.000,1.000], [0.000,0.000,2.000], [107.000,161.000,1.000], [0.000,0.000,2.000], [74.000,165.000,1.000], [63.000,225.000,1.000], [76.000,240.000,1.000], [81.000,325.000,1.000], [94.000,342.000,1.000], [109.000,390.000,0.000], [140.000,413.000,1.000], [80.000,391.000,0.000], [92.000,416.000,1.000], [0.000,0.000,2.000], [0.000,0.000,2.000], [0.000,0.000,2.000], [0.000,0.000,2.000] - In step f, using command below: > python2 c_generateLmdbs.py 2. Generate LMDB files (foot dataset) 1. Follow the steps below: ![](https://i.imgur.com/aJarPLl.png) + step a: Use the COCO dataset downloaded before. The foot annotation json files should be downloaded by yourself([link](https://cmu-perceptual-computing-lab.github.io/foot_keypoint_dataset/)). It contains 23(original 17 + 6 foot keypoints) body parts in "keypoints" + step b: + Modify line 32 and 33 to enable foot option. + Modify line 83 and 87 to match the foot annotation files we download. + step c: + Modify line 15 and 17 + step d: > python2 c_generateLmdbs.py 3. Generate LMDB files using MPII dataset ([link](http://human-pose.mpi-inf.mpg.de/#download)) 1. Follow the steps below: ![](https://i.imgur.com/I8SK4Gc.png) - For original MPII annotation (x, y, is_visible), - is_visible=0 or []: labeled but not visible - is_visible=1: labeled and visible - For refined json file, - v=0: labeled but not visible - v=1: labeled and visible - v=2: not labeled (in which case x=y=0), - ex. 070755336.jpg ![](https://i.imgur.com/p8XScfh.png) - keypoints - joint id (0 - r ankle, 1 - r knee, 2 - r hip, 3 - l hip, 4 - l knee, 5 - l ankle, 6 - pelvis, 7 - thorax, 8 - upper neck, 9 - head top, 10 - r wrist, 11 - r elbow, 12 - r shoulder, 13 - l shoulder, 14 - l elbow, 15 - l wrist) - original - [{"x":181,"y":303,"id":6,"is_visible":true}, {"x":166,"y":150,"id":7,"is_visible":false}, {"x":167.546,"y":132.3504,"id":8,"is_visible":[]}, {"x":176.454,"y":30.6496,"id":9,"is_visible":[]}, {"x":305,"y":453,"id":0,"is_visible":false}, {"x":301,"y":355,"id":1,"is_visible":true}, {"x":160,"y":317,"id":2,"is_visible":true}, {"x":201,"y":289,"id":3,"is_visible":true}, {"x":332,"y":291,"id":4,"is_visible":false}, {"x":342,"y":395,"id":5,"is_visible":false}, {"x":253,"y":251,"id":10,"is_visible":true}, {"x":173,"y":259,"id":11,"is_visible":true}, {"x":138,"y":155,"id":12,"is_visible":true}, {"x":194,"y":145,"id":13,"is_visible":false}, {"x":199,"y":213,"id":14,"is_visible":false}, {"x":258,"y":239,"id":15,"is_visible":false}] - refined - [[305.0, 453.0, 0.0], [301.0, 355.0, 1.0], [160.0, 317.0, 1.0], [201.0, 289.0, 1.0], [332.0, 291.0, 0.0], [342.0, 395.0, 0.0], [181.0, 303.0, 1.0], [166.0, 150.0, 0.0], [167.546, 132.3504, 0.0], [176.454, 30.6496, 0.0], [253.0, 251.0, 1.0], [173.0, 259.0, 1.0], [138.0, 155.0, 1.0], [194.0, 145.0, 0.0], [199.0, 213.0, 0.0], [258.0, 239.0, 0.0]] ### Image Preprocessing Problem 1.When executing a3_coco_matToMasks.m, the error "Unable to open file **"/openpose-train/openpose_train/dataset/COCO/cocoapi/images/segmentation2017/train2017/xxxxxxxxxx.jpg"** for writing. You may not have write permission." + Sol: At first, I suppose that it's the problem of writing permission of MATLAB folder. So I re-install the entire matlab under /home/cmw/. However, the same error still exists. :sweat::sweat: Then I modify the permission of MATLAB and openpose-train from 755 to 775 and nothing happens again.:sob::sob: Finally, I find that there is nothing under segmentation2017 (folder train2017 should be there). So I manually create train2017 folder and guess what? The problem is solved.:laughing::laughing: ### Generate LMDB files problems ```python= ＃因為caffe..是安裝python2.7版本，所以這邊也用python2 python2 c_generateLmdbs.py ``` 1. ![](https://i.imgur.com/DeX54Mf.png) - Sol: If the data type is 'str', using index to access it will return a str type result. **It's a version problem between python2 and python3 .** - [[Ref]](https://www.cnblogs.com/lshedward/p/9926150.html) - Ex. ```python= >>> a = "COCO" >>> type(a) >>> type(a[0]) >>> b = b'COCO' >>> type(b) >>> type(b[0]) ``` 2. ![](https://i.imgur.com/LSGDnru.png) - Sol: [[Stackoverflow solution]](https://stackoverflow.com/questions/43805999/python3-and-not-is-python2-typeerror-wont-implicitly-convert-unicode-to-byt) ### Caffe Installation (Please use python2.7) ![](https://i.imgur.com/S1iIrEk.png) #### [參考連結](https://blog.csdn.net/oJiMoDeYe12345/article/details/72900948) 1. “fatal error: hdf5.h: 没有那个文件或目录” 2. nccl.hpp:5:18: fatal error: nccl.h: No such file or directory 3. error: ‘accumulate’ is not a member of ‘std’ ![](https://i.imgur.com/aMdegsX.png) + Sol:https://stackoverflow.com/questions/7899237/function-for-calculating-the-mean-of-an-array-double-using-accumulate 4. recipe for target '.build_debug/lib/libcaffe.so.1.0.0' failed ![](https://i.imgur.com/MRThIGL.png) + Sol: - Uncomment `OPENCV_VERSION := 3` - ![](https://i.imgur.com/LnsH1sC.png) 5. ![](https://i.imgur.com/hgvh6fT.png) + Sol:Fuck no! #### 重裝 - Alright, alright, even though I solve so many problems. I CANNOT SUCCESSFULLY INSTALL THE OPENPOSETRAINCAFFE!! + Finally, I delete the entire folder and download the entire repository again. + This time, I follow the [tutorial](https://mc.ai/installing-caffe-on-ubuntu-18-04-with-cuda-and-cudnn/). For the instruction in this blog, do not execute `make clean` after successfully build. (Only use it when you want to rebuild) * If some error about missing of package were shown, just use pip3 to install it. ### Generate the Caffe ProtoTxt and shell file for training ``` python2 d_setLayers.py ``` ![](https://i.imgur.com/inC3vv4.png) 1. [Error when use python3.6](https://github.com/CMU-Perceptual-Computing-Lab/openpose_train/issues/28) - Sol: Use python2.7 instead.(The problem) ### Resume Training 1. Modify the snapshot parameter in resume_train_pose.sh. The default path of pretrained model is ***/home/cmw/openpose-train/openposetrain/training_results/pose/model/pose_iter_668000.solverstate*** ![](https://i.imgur.com/cmoDhNh.png) 2. Then in the command line, just type: ***bash resume_train_pose.sh 0*** (generated by d_setLayers.py) to start the training with the 1 GPU (0). ## Openpose Training - Custom Dataset ### Understanding Dataset Annotation - foot dataset: **openpose自己標註的腳步資料集**，相較於原始COCO資料集所提供的17個部位點多出了6個(總共23個)，標註的方法就是在原始COCO資料集JSON檔中annotations下的keypoints中，把新增的6個部位點之x,y座標以及visibility加到最後面(如下圖) ![](https://i.imgur.com/ZB8IipZ.png) ### Understanding src code (Vscode is recommended for tracing code :thumbsup::thumbsup:) - Please follow the steps mentioned in **Openpose Training - Provided Dataset** - Some code needs to be modified, see following: - a4_coco_matToRefinedJson.m:用來產生訓練用的json檔案，此檔案會被用來產生lmdb檔案 - function reshapeKeypoints(*line 334*):COCO資料集中，針對每一個部位點會給出x,y座標以及visibility，此function用來修改visibility (詳情可見上方[Openpose Preprocessing(openpose_train.md)](#Openpose-Preprocessing-openpose_trainmd)中的說明 或者是此function中的comment也有提到) - d_setLayers.py:用來產生訓練所需的training, deploy 以及solver的.prototxt檔案，事實上前述的檔案產生是call generateProtoTxt.py這隻程式完成的，在d_setLayers.py裡面主要做的事情是去產生所有的參數(所以要增加自訂資料集時，要在這個檔案裡面改，generateProtoTxt.py只負責接參數然後產生前述的.prototxt檔案) - ex.用到的dataset, 用到的lmdb資料夾位置, network的縮寫(在generateProtoTxt.py中再改成caffe的對應layer名稱)...等等 - c_generateLmdbs.py:用來產生lmdb檔案，需要事先準備好訓練資料集的圖片以及json檔案，主要是call generateLmdbFile.py這隻檔案，但因為裡面考慮了很多資料集的情況，我們自訂資料集只需要拿類似coco foot的處理方式來做就好，因此重新改出了一個generateCustomLmdbFile.py的檔案，目前專門for foot dataset，之後再看要改哪邊 - generateCustomLmdbFile.py:看會用到json檔裡面的哪些欄位，然後標記資料時只要標記這些欄位就好 1. dataset 2. img_height 3. img_width 4. numOtherPeople 5. people_index 6. annolist_index 7. numOtherPeople 8. objpos 9. scale_provided 10. joint_self 11. joint_others(如果一張圖片含有多人，才會讀到這項資料) 13. objpos_other(如果一張圖片含有多人，才會讀到這項資料) 14. scale_provided_other(如果一張圖片含有多人，才會讀到這項資料) - 要產生lmdb也會用到mask，COCO是有segmentation的資料，而mpii沒有，所以其實可以學mpii的方法，用bbox來做mask - **接續上一點，其實也不一定要有bbox來做mask，原本會做mask是因為資料集有標記不完全的情況發生，如果是自己的資料集都有標記，其實mask就直接用和原圖相同大小，所有pixel都改成255灰階值(白色)就好** > 補充：在 COCO 裡面，如果有 segmentation的標注資料，會根據此產生 bbox - If you want to customize the number of keypoints: - 假設有要用自己標記的keypoints，也就是要用到其他的skeleton，必須要去改openpose-caffe裡面的檔案並且重新build，**如果只是用相同的標記點(例如foot dataset裡面的23 keypoints)，就只要在training.prototxt裡面的models標籤使用"COCO_25B_23"就好**，如果使用相同的skeleton，但dataset不同，那就在models標籤那邊重複就好，如圖(重複了兩次COCO_25B_23):![](https://i.imgur.com/QesOAzU.png) - openpose_Caffe_train/src/caffe/openpose/poseModel.cpp - 更改Auxiliary functions底下的function (int poseModelToIndex)以及Parameters and functions to change if new PoseModel底下的每一個array的相關參數 - openpose_Caffe_train/include/caffe/openpose/poseModel.hpp - enum class PoseModel裡面要新增自己的model(注意要加在現有model的最下面以及Size的上面，順序會影響到後面oPDataTransformer在計算channel的部份) - poseModel.cpp:定義所使用dataset對應的body part keypoints、skeleton是怎麼組成(哪些部位要相連)、最後產生的model會output出那些body parts等等的參數 - oPDataTransformer.cpp:產生G.T.(ground truth)的地方 ### Custom dataset annotation - [Annotation tool](https://github.com/jsbroks/coco-annotator) - [Annotation tool setup](https://github.com/jsbroks/coco-annotator/wiki/Getting-Started) - [Annotation tool tutorial](https://hackmd.io/@cmwchw/H1XVv7jB_) #### Prepare Training LMDB file 1. a2_coco_jsonToMat.m + Add your custom dataset ex. line 40, 89, 130, 288 + Put your custom coco-style annotation file into folder: **openpose_train/dataset/COCO/cocoapi/annotations/your_dataset.json** + File **your_dataset.mat** will be created in following path: **openpose_train/dataset/COCO/mat/your_dataset.mat** 2. a3_coco_matToMasks.m + Add your custom dataset ex. line 40, 111, 128, 184 + Put your custom dataset into folder: **openpose_train/dataset/COCO/cocoapi/images/your_dataset/** + Folder **mask2017** and **segmentation2017** will be created. (Folder names don't matter, and they can be changed by code.) > **Caution**: According to line 155 in src code, the annotation will be dropped if the annotated keypoints are less than 5. 3. a4_coco_matToRefinedJson.m + Add your custom dataset ex. line 24, 136, 241 + File your_dataset.json will be created in following path: **openpose_train/dataset/COCO/json/your_dataset.json** 4. c_generateLmdbs.py + Add your custom dataset ex. line 33, 161 + Make sure following files are ready 1. lmdb files: /openpose_train/dataset/your_dataset/![](https://i.imgur.com/Uc1qvvq.png) 2. images: /openpose_train/dataset/COCO/cocoapi/images/![](https://i.imgur.com/ljOLz2z.png) 3. annotation json file: /openpose_train/dataset/COCO/json/your_dataset.json![](https://i.imgur.com/vOUFydZ.png) + Folder **lmdb_your_dataset** will be created 2 files are inside![](https://i.imgur.com/pI6E8tc.png) ## Openpose with tracking [Project Page](https://cmu-perceptual-computing-lab.github.io/spatio-temporal-affinity-fields/) [Github](https://github.com/soulslicer/openpose/tree/staf) 1. Using the same method which applied in building openpose, remember to switch to the **staf** branch. > git clone https://github.com/soulslicer/openpose.git -b staf 3. Currently, only c++ version is available. Try command below: `./openpose.bin --model_pose BODY_21A --tracking 1 --render_pose 1 --video your_video` 3. Try to use python API: ![](https://i.imgur.com/XyUjqCO.png) May be solved by links as belows: - https://github.com/soulslicer/openpose/issues/5 Go to this issue and check **sh0w**'s' comment. sh0w modifies ^1.^the pybind part to solve the data type conversion error and ^2.^enable the tracking flag. - For first modification. ***Check the [link](https://github.com/sh0w/openpose/blob/41ee8a0621f2b44be1afa3b9c9283abf8b880a65/python/openpose/openpose_python.cpp#L436) and see line 436.*** - For second modification. ***Check the [link](https://github.com/sh0w/openpose/blob/41ee8a0621f2b44be1afa3b9c9283abf8b880a65/python/openpose/openpose_python.cpp#L154) and see line 154.*** - https://github.com/CMU-Perceptual-Computing-Lab/openpose/issues/1162 ## Openpose C++ API Extension https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/04_cpp_api.md https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/examples/user_code/README.md ## Caffe Draw Loss Curve - The filepath - caffe-master/tools/extra/parse_log.sh - caffe-master/tools/extra/extract_seconds.py - caffe-master/tools/extra/plot_training_log.py.example - Parse training log ```cmd ./parse_log.sh your.log ``` - Draw curve ```cmd python2 plot_training_log.py.example yourflag yourimg.png your.log ``` - yourflag ```cmd Notes: 1. Supporting multiple logs. 2. Log file name must end with the lower-cased ".log". Supported chart types: 0: Test accuracy vs. Iters 1: Test accuracy vs. Seconds 2: Test loss vs. Iters 3: Test loss vs. Seconds 4: Train learning rate vs. Iters 5: Train learning rate vs. Seconds 6: Train loss vs. Iters 7: Train loss vs. Seconds ```