# ViSP安裝、Nerf重建與YOLOv7訓練
>[name=郭立智][name=陳英豪][time=2025_01_17]
[TOC]
## 設備規格
1. ubuntu 20.04 / 22.04
2. [ROS noetic](https://wiki.ros.org/noetic/Installation/Ubuntu) / [ROS2 humble](https://docs.ros.org/en/humble/Installation/Ubuntu-Install-Debians.html)
3. Cuda 11.3
4. [cuDNN 8.5](https://blog.csdn.net/u014297502/article/details/126863549)
5. Anaconda
With the default parameters, on a 640 x 480 image, initial pose estimation takes around 2 seconds on an Nvidia Quadro RTX 6000.
[On the same setup, a pose update (refinement) iteration takes around 60-70 milliseconds.](https://visp-doc.inria.fr/doxygen/visp-daily/tutorial-tracking-megapose.html?utm_source=chatgpt.com)
Your GPU VRAM must be bigger than 9GB due to [Out of memory issue while running megapose6d #53](https://github.com/megapose6d/megapose6d/issues/53)
## Nvidia 驅動安裝
```
$ sudo apt update && sudo apt upgrade -y
$ ubuntu-drivers devices
$ sudo ubuntu-drivers autoinstall
$ sudo reboot # 重新開機
$ nvidia-smi # 檢查是否安裝成功
```
`Terminal Output`
:::info
- 如果使用autoinstall後無法啟動，可能是ubuntu版本低且Nvidia顯示卡過新導致異常，需使用手動安裝指定版本(下為nvidia-driver-535，可自行變更)
[Nvidia 驅動](https://github.com/lutris/docs/blob/master/InstallingDrivers.md#amd--intel)
```
sudo add-apt-repository ppa:graphics-drivers/ppa && sudo dpkg --add-architecture i386 && sudo apt update && sudo apt install -y nvidia-driver-535 libvulkan1 libvulkan1:i386
```
:::
- 安裝gcc
```
$ sudo apt-get install manpages-dev
$ sudo apt-get update
$ sudo apt install build-essential
$ sudo apt-get install manpages-dev
$ gcc --version #檢查是否安裝成功
```
`Terminal Output`
## [Cuda 11.3](https://developer.nvidia.com/cuda-11.3.0-download-archive)安裝
```
wget https://developer.download.nvidia.com/compute/cuda/11.3.0/local_installers/cuda_11.3.0_465.19.01_linux.run
sudo sh cuda_11.3.0_465.19.01_linux.run --override
```
**不要安裝驅動(示意圖)**
- .bashrc新增以下程式並重啟終端機
```
# cuda
export PATH=$PATH:/usr/local/cuda/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64
export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/cuda/lib64
# cuda
```
- 使用nvcc -V查看安裝的CUDA版本
```
$ nvcc -V #檢查是否安裝成功
```
`Terminal Output`
:::info
在選擇安裝中，如果沒有提供22.04的cuda安裝包，我們可以去下載安裝20.04的包，cuda向下相容。
:::
## 安裝[cuDNN8.5](https://developer.nvidia.com/cudnn-archive)
- [官往安裝步驟](https://docs.nvidia.com/deeplearning/cudnn/latest/installation/overview.html)
```
# 到下載的資料夾
$ sudo cp cuda/include/cudnn*.h /usr/local/cuda/include
$ sudo cp -P cuda/lib64/libcudnn* /usr/local/cuda/lib64
$ sudo chmod a+r /usr/local/cuda/include/cudnn*.h /usr/local/cuda/lib64/libcudnn*
```
- 驗證安裝
```
$ git clone https://github.com/johnpzh/cudnn_samples_v8.git
$ cd $HOME/cudnn_samples_v8/mnistCUDNN
$ sudo apt install libfreeimage3 libfreeimage-dev
$ sudo apt-get install gcc-9
$ sudo apt-get install g++-9
$ sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-9 60 --slave /usr/bin/g++ g++ /usr/bin/g++-9
$ make clean && make
$ ./mnistCUDNN
```
- 執行./mnistCUDNN，出現以下畫面代表驗證成功(Test passed!)
`Terminal Output`
- 檢查cuDNN
```
$ cat /usr/local/cuda/include/cudnn_version.h | grep CUDNN_MAJOR -A 2 # 檢查是否安裝成功
```
`Terminal Output`
:::warning
1.遇到FreeImage.h找不到，可直接安裝其apk
```
mk@mk-ROG-Strix-G614JZ-G614JZ:~/cudnn_samples_v8/mnistCUDNN$ make clean && make
...
test.c:1:10: fatal error: FreeImage.h: No such file or directory
1 | #include "FreeImage.h"
| ^~~~~~~~~~~~~
compilation terminated.
>>> WARNING - FreeImage is not set up correctly. Please ensure FreeImage is set up correctly. <<<
...
```
> $ sudo apt install libfreeimage3 libfreeimage-dev
2.gcc版本過高的，解決方法
```
$ sudo apt-get install gcc-9
$ sudo apt-get install g++-9
$ sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-9 60 --slave /usr/bin/g++ g++ /usr/bin/g++-9
```
:::
## [Anaconda](https://docs.anaconda.com/anaconda/install/linux/)
- 切換到安裝腳本所在的目錄
```
$ sh Anaconda3-2023.03-1-Linux-x86_64.sh
```
- check Anaconda
```
$ source ~/.bashrc
```
```
$ conda config --set auto_activate_base True
The base environment is activated by default
$ conda config --set auto_activate_base False
The base environment is not activated by default
```
## ViSP X ROS / ROS2
- [Visp X ROS](https://hackmd.io/@TPoqXDxdQfS_CcCr6i1q4g/B14GZnWDp)
### Install prerequisites
- 安裝 GNU g++ 編譯器、CMake、git 和 subversion
```
$ sudo apt-get install build-essential cmake-curses-gui git subversion wget
```
### Create a workspace
```
$ echo "export VISP_WS=$HOME/visp-ws" >> ~/.bashrc
$ source ~/.bashrc
$ mkdir -p $VISP_WS
```
### Quick ViSP installation
- Install 3rd parties
- without nlohmann-json3-dev because megapose have to build nlohmann-json from source.
```
sudo apt-get install libopencv-dev libx11-dev liblapack-dev
sudo apt-get install libeigen3-dev libv4l-dev libzbar-dev libpthread-stubs0-dev libdc1394-dev
```
- Get ViSP source code
```
$ cd $VISP_WS
$ git clone https://github.com/lagadic/visp.git
```
### Install 3rd party megapose need
[Tutorials > Tracking > Tutorial: Tracking with MegaPose](https://visp-doc.inria.fr/doxygen/visp-daily/tutorial-tracking-megapose.html)
### JSON for modern C++
- Installation from source
If the package is not available for your distribution, you can install JSON from source:
```
$ cd $VISP_WS/visp/3rdparty
$ git clone https://github.com/nlohmann/json.git
$ cd json && mkdir build && cd build
$ cmake ..
$ make -j$(nproc) && sudo make install
$ apt list --installed | grep -i opencv
If this command does not return an empty line, please run (if you are sure that it is not required by another software installed on your computer):
$ sudo apt remove libopencv-dev
```
- Install OpenCV dependencies. On a Debian distribution, you would run:
```
$ sudo apt update
$ sudo apt install libgtk-3-dev \
cmake \
git \
pip \
cmake-curses-gui \
locate \
libx11-dev
```
- 下載opencv程式碼
```
$ cd ${HOME}/visp-ws/visp/3rdparty/
$ git clone https://github.com/opencv/opencv_contrib -b 4.7.0
$ git clone https://github.com/opencv/opencv -b 4.7.0
```
- 編譯
```
mkdir -p ${HOME}/visp-ws/visp/3rdparty/opencv/build && cd ${HOME}/visp-ws/visp/3rdparty/opencv/build
# RTX4080 pc do not run this command(Optional, Not necessary)
export GPU_CAPABILITIES=$(nvidia-smi --query-gpu=compute_cap --format=csv,noheader)
cmake .. \
-DCMAKE_BUILD_TYPE=RELEASE \
-DCMAKE_INSTALL_PREFIX=/usr \
-DCMAKE_INSTALL_LIBDIR=lib \
-DWITH_CUDA=ON \
-DWITH_CUDNN=ON \
-DOPENCV_DNN_CUDA=ON \
-DENABLE_FAST_MATH=1 \
-DCUDA_FAST_MATH=1 \
-DCUDA_ARCH_BIN=${GPU_CAPABILITIES} \
-DWITH_CUBLAS=1 \
-DOPENCV_EXTRA_MODULES_PATH=${VISP_WS}/visp/3rdparty/opencv_contrib/modules \
-DBUILD_PERF_TESTS=Off \
-DBUILD_TESTS=Off \
-DBUILD_EXAMPLES=Off \
-DBUILD_opencv_apps=Off \
-DBUILD_opencv_java_bindings_generator=Off \
-DBUILD_opencv_js=Off
-DOPENCV_GENERATE_PKGCONFIG=ON (產生.pc檔)
```
- 安裝
```
$ make -j$(nproc)
$ sudo make install
```
:::danger
```
$ sudo apt install gcc-10 g++-10
$ export CC=/usr/bin/gcc-10
$ export CXX=/usr/bin/g++-10
$ export CUDA_ROOT=/usr/local/cuda
$ ln -s /usr/bin/gcc-10 $CUDA_ROOT/bin/gcc
$ ln -s /usr/bin/g++-10 $CUDA_ROOT/bin/g++
```
:::
* 設定 PKG_CONFIG_PATH（暫時）& set system opencv4.7
```
# （暫時）
# 檢查 .pc 檔案是否產生
ls /usr/lib/pkgconfig/opencv4.pc
# 設定 PKG_CONFIG_PATH（暫時）
export PKG_CONFIG_PATH=/usr/lib/pkgconfig:$PKG_CONFIG_PATH
pkg-config --modversion opencv4
should see 4.7.0/4.10.0
（可選）永久設定（加到 ~/.bashrc）
echo 'export PKG_CONFIG_PATH=/usr/lib/pkgconfig:$PKG_CONFIG_PATH' >> ~/.bashrc
source ~/.bashrc
# system opencv4.7
✅ 額外建議（確認你用的 OpenCV headers 是正確版本）
pkg-config --cflags opencv4
應該包含 /usr/include/opencv4
確認不是走 /usr/include/x86_64-linux-gnu/opencv4（那是舊的）
# ~/.add below to ~/.bashrc for system opencv4.7
# === ViSP Workspace ===
export VISP_WS=$HOME/visp-ws
export VISP_DIR=$VISP_WS/visp-build
# === OpenCV 4.7 (from ViSP) ===
export OpenCV_DIR=$VISP_WS/visp/3rdparty/opencv/build
export PKG_CONFIG_PATH=$OpenCV_DIR/unix-install:$PKG_CONFIG_PATH
export PYTHONPATH=$OpenCV_DIR/lib/python3:$PYTHONPATH
# === ROS (keep at the front), plus OpenCV 4.7
export CMAKE_PREFIX_PATH=/opt/ros/noetic:$OpenCV_DIR:$CMAKE_PREFIX_PATH
# or just add below
export OpenCV_DIR="$HOME/visp-ws/visp/3rdparty/opencv/build"
export CMAKE_PREFIX_PATH="$OpenCV_DIR:/opt/ros/noetic:$CMAKE_PREFIX_PATH"
export PKG_CONFIG_PATH="$OpenCV_DIR/unix-install:$PKG_CONFIG_PATH"
export LD_LIBRARY_PATH="$OpenCV_DIR/lib:$LD_LIBRARY_PATH"
```
### Create a build folder and build ViSP
- 編譯
```
$ mkdir -p $VISP_WS/visp-build
$ cd $VISP_WS/visp-build
$ cmake ../visp
```
簡查是否安裝成功
```
$ cd $VISP_WS/visp-build
$ grep "To be built" ViSP-third-party.txt
$ grep "json" ViSP-third-party.txt
```
- 安裝
> $ make -j$(nproc)
新增環境變數
```
$ echo "export VISP_DIR=$VISP_WS/visp-build" >> ~/.bashrc
$ source ~/.bashrc
```
### Installing the megapose server
- Once you have configured these variables: run the installation script with:
> $ cd $VISP_WS/visp/script/megapose_server
- change to ros noetic python version
```
$ gedit megapose_environment.yml
python=3.9 -> python=3.8
$ python3 install.py
$ mkdir -p $VISP_WS/visp/script/megapose_server/megapose6d/data/megapose-models
$ python3 install.py
$ cp $VISP_WS/visp/script/megapose_server/megapose_variables_final.json ${HOME}/anaconda3/envs/megapose/lib/python3.8/site-packages/megapose_server
```
- change to ros2 humble python version
```
$ gedit megapose_environment.yml
python=3.9 -> python=3.10
$ mkdir -p $VISP_WS/visp/script/megapose_server/megapose6d/data/megapose-models
$ python3 install.py
$ cp $VISP_WS/visp/script/megapose_server/megapose_variables_final.json ${HOME}/anaconda3/envs/megapose/lib/python3.10/site-packages/megapose_server
```
## ViSP執行
### Server
```
$ conda activate megapose
(megapose) $ cd ${VISP_WS}/visp-build/tutorial/tracking/dnn
(megapose) $ unset LD_LIBRARY_PATH
(megapose) $ python -m megapose_server.run --host 127.0.0.1 --port 5555 --model RGB --meshes-directory data/models
```
### Client
```
$ cd $VISP_WS/visp-build/tutorial/tracking/dnn
# videos demo
$ ./tutorial-megapose-live-single-object-tracking --config data/megapose_cube.json megapose/address 127.0.0.1 megapose/port 5555 video-device data/cube_video.mp4
# realsense demo
sudo apt install v4l-utils
v4l2-ctl --list-devices
./tutorial-megapose-live-single-object-tracking --config data/megapose_leaf.json megapose/address 127.0.0.1 megapose/port 5555 video-device /dev/video8
```
## ROS執行
### 下載與編譯
```
$ cd path/to/your/workspace/src
$ git clone https://github.com/Yuntechec404/vision_visp -b ros1-noetic
$ cd ..
$ sudo apt-get install ros-noetic-cv-bridge*
$ sudo apt install ros-noetic-camera-calibration-parsers*
$ catkin_make
$ conda activate megapose
(megapose)$ pip install rospkg rospy catkin_tools
(megapose)$ pip install transforms3d
```
### Server
```
$ gedit ${HOME}/catkin_ws/devel/lib/visp_megapose/megapose_server_.py
#!/usr/bin/python3 -> #!/home/user/anaconda3/envs/megapose/bin/python3
$ cd path/to/your/workspace/
~/workspace$ conda activate megapose
(megapose) ~/workspace$ export LD_LIBRARY_PATH=${HOME}/anaconda3/envs/megapose/lib:$LD_LIBRARY_PATH
(megapose) ~/workspace$ source devel/setup.bash
(megapose) ~/workspace$ roslaunch visp_megapose megapose_server.launch
```
### Client
```
$ cd path/to/your/workspace/
~/workspace$ source devel/setup.bash
~/workspace$ roslaunch visp_megapose megapose_client.launch
```
:::info
[Megapose(ViSP) 移植ROS 遇到的問題](https://hackmd.io/@sdJzFAKlRceXiJYdeWkJ0g/SJf0WPzLyg)
:::
## ROS2執行
### 下載與編譯
```
$ cd path/to/your/workspace/src
$ git clone https://github.com/Yuntechec404/vision_visp
$ git clone https://github.com/ros-perception/vision_opencv.git
$ cd ..
$ colcon build --packages-select cv_bridge
$ colcon build --packages-select visp_bridge
$ colcon build --packages-select visp_megapose
$ conda activate megapose
(megapose)$ pip3 install transforms3d
```
### Server
```
$ conda activate megapose
(megapose)$ cd path/to/your/workspace/src
(megapose)~/workspace$ unset LD_LIBRARY_PATH
(megapose)~/workspace$ source install/setup.bash
(megapose)~/workspace$ ros2 launch visp_megapose megapose_server.launch
```
### Client
```
$ cd path/to/your/workspace/src
~/workspace$ LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${HOME}/visp-ws/visp-build/lib
~/workspace$ source install/setup.bash
~/workspace$ ros2 launch visp_megapose megapose_client.launch
```
## NeRF重建物體3D模型
:::warning
:warning: 安裝NeRFstudio 需要安裝CUDA11.7 or 11.8
:::
[NeRFstudio安裝流程 ](https://docs.nerf.studio/quickstart/installation.html)
### 安裝[CUDA11.8](https://developer.nvidia.com/cuda-11-8-0-download-archive)
```
$ sudo apt-get install nvidia-driver-560
$ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-ubuntu2204.pin
$ sudo mv cuda-ubuntu2204.pin /etc/apt/preferences.d/cuda-repository-pin-600
$ wget https://developer.download.nvidia.com/compute/cuda/11.8.conda create --name nerfstudio -y python=3.80/local_installers/cuda-repo-ubuntu2204-11-8-local_11.8.0-520.61.05-1_amd64.deb
$ sudo dpkg -i cuda-repo-ubuntu2204-11-8-local_11.8.0-520.61.05-1_amd64.deb
$ sudo cp /var/cuda-repo
-ubuntu2204-11-8-local/cuda-*-keyring.gpg /usr/share/keyrings/
$ sudo apt-get update
$ sudo apt-get -y install cuda-11-8
```
- .bashrc新增以下程式並重啟終端機
```
# cuda
export PATH=$PATH:/usr/local/cuda/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64
export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/cuda/lib64
# cuda
```
- 使用nvcc -V查看安裝的CUDA版本
```
$ nvcc -V #檢查是否安裝成功
```
### 建立虛擬環境
- 安裝Anaconda(看上面)
```
$ conda create --name nerfstudio -y python=3.8
$ conda activate nerfstudio
(nerfstudio) $ python -m pip install --upgrade pip
### PyTorch 2.1.2
(nerfstudio) $ pip install torch==2.1.2+cu118 torchvision==0.16.2+cu118 --extra-index-url https://download.pytorch.org/whl/cu118
```
:::info
如果安裝了 2.0.1 之前的版本，則應卸載先前版本的 pytorch、functorch 和tiny-cuda-nn。
```
$ pip uninstall torch torchvision functorch tinycudann
```
:::
- 透過cuda建置cuda-toolki
```
(nerfstudio) $ conda install -c "nvidia/label/cuda-11.8.0" cuda-toolkit
```
- 安裝 tiny-cuda-nn/gsplat
```
(nerfstudio) $ pip install ninja git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
```
### 安裝NeRFstudio
```
(nerfstudio) $ pip install nerfstudio
(nerfstudio) $ git clone https://github.com/nerfstudio-project/nerfstudio.git
(nerfstudio) $ cd nerfstudio
(nerfstudio)~/nerfstudio$ pip install --upgrade pip setuptools
(nerfstudio)~/nerfstudio$ pip install -e .
```
- Tab 補全（選配，使用時更方便）
```
(nerfstudio)~/nerfstudio$ ns-install-cli
(nerfstudio)~/nerfstudio$ pip install -e .[dev]
(nerfstudio)~/nerfstudio$ pip install -e .[docs]
```
### NeRFstudio Demo
```
#Download some test data:
(nerfstudio)~/nerfstudio$ ns-download-data nerfstudio --capture-name=poster
#Train model
(nerfstudio)~/nerfstudio$ ns-train nerfacto --data data/nerfstudio/poster
```
### 產生數據集
- [Tutorial: Exporting a 3D model to MegaPose after reconstruction with NeRF](https://visp-doc.inria.fr/doxygen/visp-daily/tutorial-megapose-model.html)
- [3D模型製作COLMAP](https://hackmd.io/@sdJzFAKlRceXiJYdeWkJ0g/BJ4qdc_cA)
- [3D模型製作Spectacular AI](https://hackmd.io/@sdJzFAKlRceXiJYdeWkJ0g/H1BJKcOqR)
### Nerf 訓練
[指令說明](https://github.com/nerfstudio-project/nerfstudio/issues/1800)
```
$ conda activate nerfstudio
(nerfstudio)~/nerfstudio$ NERF_DIR=/training_nerf/save/to/path
(nerfstudio)~/nerfstudio$ ns-train nerfacto --pipeline.model.predict-normals True --data $NERF_DATA_DIR --output-dir $NERF_DIR
```
### Gaussian 訓練
[指令說明](https://docs.nerf.studio/nerfology/methods/splat.html)
```
$ conda activate nerfstudio
(nerfstudio)~/nerfstudio$ NERF_DIR=/training_nerf/save/to/path
(nerfstudio)~/nerfstudio$ ns-train splatfacto --data $NERF_DATA_DIR --output-dir $NERF_DIR
```
:::danger
`Terminal`
```
...
warnings.warn(
( ● ) gsplat: Setting up CUDA with MAX_JOBS=10 (This may take a few minutes
the first time)
Killed
```
通常是由於系統資源不足，特別是內存或顯存不足導致的。
```
$ export MAX_JOBS=2
```
:::
### ns-viewer檢視Nerf結果
```
$ conda activate nerfstudio
(nerfstudio)~/nerfstudio$ ns-viewer --load-config <path_to_your_trained_model_config.yaml>
```
### Converting NeRF representation to .obj / .ply
- 使用ns-viewer並照步驟設定
1. 切到 Export
2. Use Crop
3. 確認裁切是否正確
4. 將其指令複製(Point Cloud是ply檔，Mesh是obj檔)
5. 將ns-viewer或train-NeRF關閉(因Cuda記憶體會不足須將訓練關閉)
```
$ conda activate nerfstudio
(nerfstudio)$ cd nerfstudio
(nerfstudio)~/nerfstudio$ 執行複製的指令
```
:::info
如執行錯誤大多原因為以下路徑問題
IMAGES_DIR,NERF_DATA_DIR,NERF_DIR,NERF_MODEL_DIR
:::
:::info
此為ViSP提供方法，如果背景過於複雜可能會不太理想，建議使用ns-viewer內建功能
```
(nerfstudio)~/nerfstudio$ NERF_MODEL_DIR=/training_model/save/to/path
(nerfstudio)~/nerfstudio$ ns-export poisson --load-config ${NERF_DIR}/nerfacto/2023-06-13_171130/config.yml --output-dir $NERF_MODEL_DIR
```
- 轉檔成功
:::
### Importing the model in blender
- 將轉檔後的.obj匯入Blender 3D視覺軟體中做修圖
- 匯入.obj與.mtl檔案
### 進行模型預測
- 開啟伺服器(Starting the server)
```
$ conda activate megapose
(megapose) $ cd $VISP_WS/visp-build/tutorial/tracking/dnn
(megapose) $ python -m megapose_server.run --host 127.0.0.1 --port 5555 --model RGB --meshes-directory data/models
```
- 開始進行模型預測，json檔會因為模型命名而改要更改，video-device後面的0也會因為實際設備讀的port不同而要更改。
## Yolov7
- [官網說明](https://visp-doc.inria.fr/doxygen/visp-daily/tutorial-synthetic-blenderproc.html)
### 安裝 Blenderproc
- 建立一個新的 conda 環境，以避免與其他 Python 套件發生潛在衝突。
```
$ conda create --name blenderproc python=3.10 pip
$ conda activate blenderproc
$ pip install blenderproc
```
- 下載Blender並設定環境，此設定僅執行一次。
```
(blenderproc) $ blenderproc quickstart
```
- 以下命令檢查其輸出
```
(blenderproc) $ blenderproc vis hdf5 output/0.hdf5
```
- 模擬器需提供有用的數據，應該獲得真實的紋理，可以從 cc0textures.com 下載材質包，其中包含 1500 多種高解析度材質。
```
(blenderproc) $ blenderproc download cc_textures path/to/folder/where/to/save/materials/
```
:::info
建議接網路線。
下載完整資料集可能會佔用大量磁碟空間（30+GB）。您可以安全地刪除一些材料或在腳本獲取足夠的材料後停止腳本。雖然在執行快速測試時使用少量材料可能很有用，但應優先使用全套材料，因為當將深度學習模型轉移到現實世界數據時，多樣性會有所幫助。
:::
### 生成數據集
```
$ cd /home/path/to/visp-ws/visp/script/dataset_generator
```
- 開啟example_config.json 並修改相應參數[詳情](https://visp-doc.inria.fr/doxygen/visp-daily/tutorial-synthetic-blenderproc.html)
```json=
{
"numpy_seed": 19,
"blenderproc_seed": "69",
"models_path": "/home/mk/models", //模型路徑
"cc_textures_path": "/home/mk/cc_textures", //材料包路徑
"camera": {
"px": 600,
"py": 600,
"u0": 320,
"v0": 240,
"h": 480,
"w": 640,
"randomize_params_percent": 5.0
},
"rendering": {
"max_num_samples": 32,
"denoiser": "OPTIX"
},
"scene": {
"room_size_multiplier_min": 5.0,
"room_size_multiplier_max": 10.0,
"simulate_physics": false,
"max_num_textures": 50,
"distractors": {
"min_count": 20,
"max_count": 50,
"min_size_rel_scene": 0.05,
"max_size_rel_scene": 0.1,
"custom_distractors": null, //遮蔽物腳本(無輸入 null)
"custom_distractor_proba": 0.5,
"displacement_max_amount": 0.0,
"pbr_noise": 0.5,
"emissive_prob": 0.0,
"emissive_min_strength": 2.0,
"emissive_max_strength": 5.0
},
"lights": {
"min_count": 3,
"max_count": 6,
"min_intensity": 50,
"max_intensity": 200
},
"objects": {
"min_count": 2,
"max_count": 5,
"multiple_occurences": true,
"scale_noise": 0.2,
"displacement_max_amount": 0.0,
"pbr_noise": 0.3,
"cam_min_dist_rel": 1.0,
"cam_max_dist_rel": 3.0
}
},
"dataset": {
"save_path": "./output_blenderproc", //輸出路徑
"scenes_per_run": 1,
"num_scenes": 200,
"images_per_scene": 20,
"empty_images_per_scene": 2, //總資料 = (images_per_scene + empty_images_per_scene) * num_scenes
"pose": true,
"depth": false,
"normals": false,
"segmentation": false,
"detection": true,
"detection_params": {
"min_side_size_px": 10,
"min_visibility_percentage": 0.3,
"points_sampling_occlusion": 100
}
}
}
```
- 運行generate_dataset.py生成資料集，完成後確認輸出路徑與影像數量
```
(blenderproc)~/path/to/visp/script/dataset_generator $ python generate_dataset.py --config path/to/config.json
```
:::info
在Quadro RTX 6000上產生1000張影像（解析度為640 x 480）並偵測單個物體大約需要30分鐘。
:::
:::danger
3D model format:
可同時生成多個模型資料集，格式如下。
...
"models_path": "/home/mk/.../data/models",
...
:::
### yolov7下載
```
$ git clone https://github.com/WongKinYiu/yolov7.git
$ cd yolov7
~/yolov7 $ conda create --name yolov7 python=3.10 pip
~/yolov7 $ conda activate yolov7
```
### 安裝[PyTorch](https://pytorch.org/get-started/locally/)
根據你的CUDA版本進行安裝
```
ex:CUDA 11.8
(yolov7) ~/yolov7 $conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia
```
:::info
可能會有更新，建議進入官網複製指令
:::
- 檢查
```
(yolov7)$ python
>>> import torch
>>> print(torch.__version__)
```
`Terminal Output`
```
2.3.1
```
### 安裝依賴項
```
(yolov7) ~/yolov7 $ pip install opencv-python
(yolov7) ~/yolov7 $ wget https://github.com/WongKinYiu/yolov7/releases/download/v0.1/yolov7-tiny.pt
(yolov7) ~/yolov7 $ pip install -r requirements.txt
```
### Demo
```
(yolov7) ~/yolov7 $ python detect.py --weights yolov7.pt --conf 0.25 --img-size 640 --source inference/images/horses.jpg
```
輸出路徑/path/to/yolov7/runs/detect/exp/horses.jpg
### 訓練
- 將blenderproc資料集重新格式化為YoloV7的格式
- "input"：blenderproc資料集的路徑，
- "output"：儲存YoloV7格式的資料集資料夾。
- "train-split"：值0.8表示資料集的80%用於訓練，而20%用於驗證;分割是在所有場景中隨機執行。
```
$ cd /home/path/to/visp-ws/script/dataset_generator
(blenderproc) ~/.../dataset_generator $ python export_for_yolov7.py --input path/to/dataset --output path/to/yolodataset --train-split 0.8
```
```yaml=
names:
- pallet
nc: 1
train: /home/mk/yolov7/data/yolodataset/images/train
val: /home/mk/yolov7/data/yolodataset/images/val
```
- 複製兩個新檔案：網路配置和參數。
```
$ CFG=/path/to/yolov7/cfg/training/yolov7-tiny-custom.yaml
$ cp /path/to/yolov7/cfg/training/yolov7-tiny.yaml $CFG
$ HYP=/path/to/yolov7/data/hyp.scratch.tiny-custom.yaml
$ cp /path/to/yolov7/data/hyp.scratch.tiny.yaml $HYP
```
- 開啟新的cfg文件，並修改預訓練的類別數量(nc)。
- 調完YoloV7-tiny後，執行
```
$ YOLO_DATASET=/path/to/dataset
$ IMG_SIZE=640
$ YOLO_NAME=blenderproc-tiny
(yolov7) $ python train.py --workers 8 --device 0 --batch-size 64 --data "${YOLO_DATASET}/dataset.yaml" --img $IMG_SIZE $IMG_SIZE --cfg $CFG --weights yolov7-tiny.pt --name $YOLO_NAME --hyp $HYP
```
:::info
如果訓練期間記憶體不夠，請將batch-size調低進行訓練
:::
[yolov5训练时出现_pickle.UnpicklingError: STACK_GLOBAL requires str的解决办法](https://blog.csdn.net/weixin_44358914/article/details/120247850)
### 運行測試
- weights 權重路徑 --source 檔案路徑/相機設備
```
$ cd yolov7
~/yolov7 $ conda activate yolov7
(yolov7)~/yolov7 $ python detect.py --weights ./runs/train/blenderproc-tiny-pallet/weights/best.pt --source 2
```
```
cd $VISP_WS/visp-build/tutorial/detection/dnn
$ ./tutorial-dnn-object-detection-live --model /home/user/catkin_ws/src/vision_visp/visp_megapose/data/pallet_new.onnx --config none --type yolov7 --framework onnx --width 640 --height 640 --nmsThresh 0.5 --mean 0 0 0 --filterThresh -0.25 --scale 0.0039
```
### *.pt to *.onnx
```
$ cd yolov7
~/yolov7 $ conda activate yolov7
(yolov7) $ pip install onnx
```
- weights：指定模型權重文件的路徑。
- grid：這通常用於指定是否使用YOLO的網格結構進行導出。
- simplify：簡化模型結構，通過ONNX的模型簡化工具進行的。
- topk-all 100：指定在推理過程中選取置信度最高的100個對象。
- iou-thres 0.65：設置IOU（交並比）的閾值，可能用於非極大值抑制（NMS）。
- conf-thres 0.35：設置對象檢測的置信度閾值，低於該值的檢測結果將被忽略。
- img-size 640 640: 输入的模型圖像大小。
- max-wh 640: 檢測框的最大寬高，如果檢測框的尺寸大於此值，它将被裁剪或過濾。
```
(yolov7)~/yolov7 $ python3 export.py --weights ../weights/yolov7-tiny.pt --grid --simplify --topk-all 100 --iou-thres 0.65 --conf-thres 0.35 --img-size 640 640 --max-wh 640
```
### MegaPose導入Yolo
- 修改megapose_client.launch,detector_method,detector_model_path
```xml=
<launch>
...
<!-- select the detection method (CLICK or DNN) -->
<param name="detector_method" value="DNN"/>
<!-- set the path to the model directory, format is yolov7 onnx -->
<param name="detector_model_path" value="$(find-pkg-share visp_megapose)/data/pallet_new.onnx"/>
<!-- set the DNN detector detect object name -->
<param name="object_name" value="pallet"/>
</node>
</launch>
```
Sign in with Wallet
Connect another wallet