# Migration notes
目前有兩種系統可以使用，分別為MiniPC, Raspberry pi 4B，而這兩個系統同樣都有2種camera可以做選擇－logitech C920 和 minicam

目前已知MiniPC上可穩定執行logitech C920，此篇教學為探討在MiniPC上執行minicam和在Raspberry pi 4B上執行minicam，而Raspberry pi 4B上執行logitech C920則作為對照組
# I.MiniPC (Ubuntu 16.04)
目前可穩定執行的攝像頭為logitech C920, 但為了降低生產成本，我們在此嘗試用minicam來達到一樣的結果。 由於目前可成功在MiniPC上執行logitech C920，所以可直接移植編譯環境到新電腦上(見1.2)，或可以從頭建置編譯環境(從1.1開始)
## 1.Memo
**如果你已經有完整的可執行的編譯環境，可以直接從1.2開始。 若你是使用新電腦，或者想從頭重新建置編譯環境，則需安裝Opencv(3.1), Boost(1.58), QT(5.5), V4l2(4.4+) (也就是從1.1開始)**
### 1.1 install all (Opencv3.1, Boost 1.58, QT, V4l2 4.4+)
#### **1.1.1 OpenCV 3.1**

cvVersion="3.1"
sudo rm -rf opencv/build
sudo rm -rf opencv_contrib/build
cwd=$(pwd)
sudo apt -y update
sudo apt -y upgrade
sudo apt-get -y remove x264 libx264-dev

## Install dependencies
sudo apt-get -y install build-essential checkinstall cmake pkg-config yasm
sudo apt-get -y install git gfortran
sudo apt-get -y install libjpeg8-dev libjasper-dev libpng12-dev

sudo apt-get -y install libtiff5-dev

sudo apt-get -y install libtiff-dev

sudo apt-get -y install libavcodec-dev libavformat-dev libswscale-dev libdc1394-22-dev
sudo apt-get -y install libxine2-dev libv4l-dev
cd /usr/include/linux
sudo ln -s -f ../libv4l1-videodev.h videodev.h
cd $cwd

sudo apt-get -y install libgstreamer0.10-dev libgstreamer-plugins-base0.10-dev
sudo apt-get -y install libgtk2.0-dev libtbb-dev qt5-default
sudo apt-get -y install libatlas-base-dev
sudo apt-get -y install libmp3lame-dev libtheora-dev
sudo apt-get -y install libvorbis-dev libxvidcore-dev libx264-dev
sudo apt-get -y install libopencore-amrnb-dev libopencore-amrwb-dev
sudo apt-get -y install libavresample-dev
sudo apt-get -y install x264 v4l-utils

# Optional dependencies
sudo apt-get -y install libprotobuf-dev protobuf-compiler
sudo apt-get -y install libgoogle-glog-dev libgflags-dev
sudo apt-get -y install libgphoto2-dev libeigen3-dev libhdf5-dev doxygen
sudo apt-get -y install python3-dev python3-pip
sudo -H pip3 install -U pip numpy
sudo apt-get -y install python3-testresources
# Install virtual environment
python3 -m venv OpenCV-"$cvVersion"-py3
echo "# Virtual Environment Wrapper" >> ~/.bashrc
echo "alias workoncv-$cvVersion=\"source $cwd/OpenCV-$cvVersion-py3/bin/activate\"" >> ~/.bashrc
source "$cwd"/OpenCV-"$cvVersion"-py3/bin/activate
#############
############ For Python 3 ############
# now install python libraries within this virtual environment
pip install numpy dlib
# quit virtual environment
deactivate

git clone https://github.com/opencv/opencv.git
cd opencv
git checkout 3.1
cd ..

git clone https://github.com/opencv/opencv_contrib.git
cd opencv_contrib
git checkout 3.1
cd ..
cd opencv
mkdir build
cd build
cmake -D CMAKE_BUILD_TYPE=RELEASE \
            -D CMAKE_INSTALL_PREFIX=/usr/local \
            -D INSTALL_C_EXAMPLES=ON \
            -D INSTALL_PYTHON_EXAMPLES=ON \
            -D WITH_TBB=ON \
            -D WITH_V4L=ON \
            -D OPENCV_PYTHON3_INSTALL_PATH=$cwd/OpenCV-$cvVersion-py3/lib/python3.5/site-packages \
        -D WITH_QT=ON \
        -D WITH_OPENGL=ON \
        -D OPENCV_EXTRA_MODULES_PATH=../../opencv_contrib/modules \
        -D BUILD_EXAMPLES=ON ..
sudo make -j$(nproc)
sudo make install
echo "sudo modprobe bcm2835-v4l2" >> ~/.profile

### 1.1.2 boost 1.58

wget -O boost_1_58_0.tar.gz https://sourceforge.net/projects/boost/files/boost/1.55.0/boost_1_58_0.tar.gz/download
tar xzvf boost_1_58_0.tar.gz
cd boost_1_58_0/
sudo apt-get update
sudo apt-get install build-essential g++ python-dev autotools-dev libicu-dev build-essential libbz2-dev libboost-all-dev
./bootstrap.sh --prefix=/usr/local
./b2
sudo ./b2 install

### 1.1.3 qt (current version is QT5.5.1)

sudo apt-get install qt5-default qtcreator 

**On your first time to run qtcreator,you need to go to projects/manage kits/build&run/kits/desktop/,and choose the first compiler(GCC c,x86 in /usr/bin) , and click apply and ok.**

### 1.1.4 V4l2 (current version is 4.4.228)

sudo apt-get install autoconf gettext libtool libjpeg62-dev
git clone git://git.linuxtv.org/v4l-utils.git
cd v4l-utils
autoreconf -vfi
./configure
make
sudo make install
sudo ldconfig

---

## 1.2 Simple install (only need to install QT)
直接移植編譯環境，但QT仍需重新安裝

### 1.2.1 INSTALL QT

sudo apt-get install qt5-default qtcreator

**On your first time to run qtcreator, you need to go to Tools/Options/build&run/kits/desktop/, and choose the first compiler(GCC c,x86 in /usr/bin) , and click apply and ok.**
### 1.2.2 Copy files (Opencv 3.1, Boost 1.58, V4l2)
copy ethan's computer's /usr/local/include to your computer's /usr/local/include
copy ethan's computer's /usr/local/lib to your computer's /usr/local/lib

export LD_LIBRARY_PATH=/usr/local/lib
sudo ldconfig

### 1.2.3 Let v4l2 to be globally used
[Optional] copy v4l2-ctl bin to /usr/bin

cp v4l2-ctl /usr/bin

---

## 1.3 Put V4l2 to your build directory
when you already build,don’t forget to move lib/v4l2 executable into your build directories

sudo mv lib [your project path]/build-XXXXXX

---
## 1.4 QT plugin
當你想把QT的project搬到其他地方執行的時候，例如，我想把gui_calibration搬到dart_vision_minipc裡執行，若我只純粹搬執行檔，那就會跳出如下error，
> This application failed to start because it could not find or load the Qt platform plugin "xcb"
in "".
Available platform plugins are: eglfs, linuxfb, minimal, minimalegl, offscreen, xcb.
Reinstalling the application may fix this problem.
./run_gui_calibration.sh: line 16: 14415 Aborted (core dumped) sudo PATH=./lib:$PATH LD_LIBRARY_PATH=./lib ./gui_calibration

正確的方法是必須在編譯的資料夾內"ldd gui_calibration"，拿到所有需要的dependency，並找到你QT的plugin/platform，若不知道在哪可以在terminal裡下"locate platform"，將platform一整個資料夾複製過來，並在platform裡下"ldd libqtxcb.so"，找到所有需要的lib後放到./lib裡就可以，詳細方法可以看 [打包免安裝的QT程式](https://www.itread01.com/content/1547077157.html)

記得把project裡的parameter資料夾也搬過去
## 1.5 常用指令
欲看到攝影機支援的所有格式(frame rate，解析度等)，你可以在終端機上輸入
> v4l2-ctl --list-formats-ext
## 2. Problems:

#### 2.1 MJPEG

When I tried to set MJPG mode in opencv 3.1, it failed,
but I found it will succeed in opencv 3.4. However, our calibration and vision code are currently based on opencv 3.1, it will be a trouble when we migrate it to Raspberry Pi 4B. The following code snippet is not supported by opencv 3.1.

CAM[iCAM].set(cv::CAP_PROP_FOURCC,cv::VideoWriter::fourcc('M','J','P','G'));

> VIDEOIO ERROR: V4L: Property <unknown property string>(6) not supported by device

亦或是可以直接用v4l2-ctl控制PROP_FOURCC，但這還需再多研究。
#### 2.2 Unstable exposure
當攝影機經插拔或關機再開後(即斷電)，它就會變得比平常亮，看似曝光或亮度跑掉了，但當我用"v4l2-ctl --all"弄出來看發現，它的值竟然是跟正常的時候一樣的。
##### 2.2.1 觀察到的現象

cap.set(cv::CAP_PROP_AUTO_EXPOSURE,0.75);//auto on 
cap.set(cv::CAP_PROP_AUTO_EXPOSURE,0.25);
cap.set(cv::CAP_PROP_EXPOSURE, 0.001);//This should only work in Opencv3.4

but it only works with Opencv3.4 lib
In Opencv3.1, you can only run the following commands on your command line after you install v4l2 (the command is to enable exposure_auto)

v4l2-ctl -d /dev/video0 -c exposure_auto=3 
v4l2-ctl -d /dev/video1 -c exposure_auto=3 
v4l2-ctl -d /dev/video2 -c exposure_auto=3 

我也試過另一種方法，就是在程式的vision/vision_webcam.cpp裡用system下上面的三行v4l2-ctl指令，但你不能開了就馬上將它關起來，必須開完後再delay 10秒左右才能關起來，這方法看來是現在發現的方法裡最可行的，但缺點是必須等10秒還是有點太久。
##### 2.2 解法
在試了許多方法後，也試了在很多地方放先開再關的程式碼，我發現和響應的時間有很大的關係，最後，我在os/ostools.cpp裡增加一個函數如下

void OSTools::EnableCameraAutoExposure(const int id)
{
    std::stringstream cam_commd;
    cam_commd<< "./lib/v4l2-ctl -d"
        << camera::Map(id)
        << "-c exposure_auto=3";
    if(system(cam_commd.str.c_str())!=0)
    {
        console::Log(ERR) << GetWorkingPath() << std::endl;
        console::Log(ERR) << cam_commd_str() <<std::endl;        
    }
    std::this_thread::sleep_for(std::chrono::milliseconds(200));   
}

並在os/ostools.h裡增加一行

static void EnableCameraAutoExposure(const int id);

接著我在vision/vision_webcam_api.cc裡多加了一個function如下:

void vision_webcam_api::InitializeCameraAutoExposure(void)
{
int i;
for(i=0;i<3;i++){
    OS::OSTools::EnableCameraAutoExposure(i);  
}
for(i=0;i<3;i++){
    OS::OSTools::EnableCameraAutoExposure(i);
    OS::OSTools::SetCameraExposure(i,9);
}
}

當然也要在vision/vision_webcam_api.h裡加上

void InitializeCameraAutoExposure(void);

最後我們在"doStuffExecute()"裡呼叫這個函數

InitializeCameraAutoExposure();
while(!stop)...

這個方法我插拔試了十幾次，開關機也試了六七次都是沒有問題的，記住，usb插上後它需要時間重設，插上後等個二十到三十秒以後再跑。

---

#### 2.3 Delay
USB camera seems to let me wait longer than C920 when I open QT's window.
#### 2.4 Unable to run executable outside
##### 2.4.1 xcb problem

When I want to run gui_calibration on dart_vision_minipc,it came out this error
> This application failed to start because it could not find or load the Qt platform plugin "xcb"
in "".
Available platform plugins are: eglfs, linuxfb, minimal, minimalegl, offscreen, xcb.
Reinstalling the application may fix this problem.
./run_gui_calibration.sh: line 16: 14415 Aborted (core dumped) sudo PATH=./lib:$PATH LD_LIBRARY_PATH=./lib ./gui_calibration

solution:
這是因為你並沒有將你qt裡的plugin/platform copy到你外面的資料夾，且少了一個ldd libqxcb.so的動作，這裡的步驟比較複雜，你可以參考https://www.itread01.com/content/1547077157.html
我之前會發生這個問題是因為，我的platform需要的Library和原本的Platform需要的lib不一樣，所以你只要把你的platform copy出來再ldd拿到你的platform需要的lib，並把它copy進./lib就可以了，這個動作可以參考上述文章的copylib.sh來做會很方便。
##### 2.4.2 invalid key
當你照上述教學將code移到dart_vision_minipc上跑的時候，你可能會遇到一個error
> Error invalid key:langue.cn.menu_chinese.....

solution: 這是因為parameter不對，請務必把自己project裡的parameter資料夾複製一份放到外面的資料夾，我之前會發生這個錯是因為，parameter有被更新過，所以parameter會發生Assertion failed。

---

## 3.Todo List:

#### 3.1 Dual type of cameras consideration issue
Comply old C920 cameras with MiniCAMs.
#### 3.2 Minicam 10 fps
Our MiniCAMs need to have 10 fps options.
#### 3.3 Mass production
1. 像1.4章一樣打包gui_calibration及dart_vision
2. 像將打包完的lib放到dart_vision_minipc內
3. 將gui_calibration和dart_vision放到dart_vision_minipc內
4. 將project裡的Parameter資料夾放到dart_vision_minipc內

當我們以後想大量生產MiniCAM，我們必須把整個執行檔包成一包，使它可以直接攜帶在每台電腦上跑，不用再另外安裝任何東西，但我現在遇到的問題是，當我的執行檔已經可以在別的地方run了後，loader在我的電腦上竟然會顯示缺QT的plugin，這可能是因為我的QT的環境和ethan原本的還有點不一樣，等等把兩個lib merge起來再試試看。

---

## II.Raspberry Pi 4B (Raspbian Buster, kernel version: 4.19)
Future, we want to replace Raspberry Pi 4B with our MiniPC. Therefore, I have wrote down all processes step by step to explain how to migrate our current codes to RPi.
## 1.MEMO:
### 1.1 Installation (OpenCV 3.4, Boost 1.58, QT 5.7, v4l2 4.4.228)
#### 1.1.1 Prepare [硬體設備]
We need one Raspberry Pi 4B, a 64GB MicroSD, an official charging line, an official HDMI line, a heat sink for RPi.
#### 1.1.2 Setup your Raspberry Pi 4B(下文會用RPi表示) [以下為軟體設置]
1. Go to [Raspberry pi os download](https://www.raspberrypi.org/downloads/raspberry-pi-os/) and download Raspberry Pi OS (32-bit) with desktop and recommended software
2. Burn your Raspbian Buster into your microSD via Balena Etcher
3. Plug on your RPi and turn on. Then you need to do some configurations.
#### 1.1.3 OpenCV 3.4

sudo apt-get -y purge wolfram-engine
sudo apt-get -y purge libreoffice*
sudo apt-get -y clean
sudo apt-get -y autoremove
cvVersion="3.4.4"
rm -rf opencv/build
rm -rf opencv_contrib/build
cwd=$(pwd)
sudo apt -y update
sudo apt -y upgrade
sudo apt-get -y remove x264 libx264-dev

## Install dependencies
sudo apt-get -y install build-essential checkinstall cmake pkg-config yasm
sudo apt-get -y install git gfortran
sudo apt-get -y install libjpeg8-dev libjasper-dev libpng12-dev

sudo apt-get -y install libtiff5-dev

sudo apt-get -y install libtiff-dev

sudo apt-get -y install libavcodec-dev libavformat-dev libswscale-dev libdc1394-22-dev
sudo apt-get -y install libxine2-dev libv4l-dev
cd /usr/include/linux
sudo ln -s -f ../libv4l1-videodev.h videodev.h
cd $cwd

sudo apt-get -y install libgstreamer0.10-dev libgstreamer-plugins-base0.10-dev
sudo apt-get -y install libgtk2.0-dev libtbb-dev qt5-default
sudo apt-get -y install libatlas-base-dev
sudo apt-get -y install libmp3lame-dev libtheora-dev
sudo apt-get -y install libvorbis-dev libxvidcore-dev libx264-dev
sudo apt-get -y install libopencore-amrnb-dev libopencore-amrwb-dev
sudo apt-get -y install libavresample-dev
sudo apt-get -y install x264 v4l-utils

# Optional dependencies
sudo apt-get -y install libprotobuf-dev protobuf-compiler
sudo apt-get -y install libgoogle-glog-dev libgflags-dev
sudo apt-get -y install libgphoto2-dev libeigen3-dev libhdf5-dev doxygen
sudo apt-get -y install python3-dev python3-pip
sudo -H pip3 install -U pip numpy
sudo apt-get -y install python3-testresources
# Install virtual environment
python3 -m venv OpenCV-"$cvVersion"-py3
echo "# Virtual Environment Wrapper" >> ~/.bashrc
echo "alias workoncv-$cvVersion=\"source $cwd/OpenCV-$cvVersion-py3/bin/activate\"" >> ~/.bashrc
source "$cwd"/OpenCV-"$cvVersion"-py3/bin/activate
#############
############ For Python 3 ############
# now install python libraries within this virtual environment
sudo sed -i 's/CONF_SWAPSIZE=100/CONF_SWAPSIZE=1024/g' /etc/dphys-swapfile
sudo /etc/init.d/dphys-swapfile stop
sudo /etc/init.d/dphys-swapfile start
pip install numpy dlib
# quit virtual environment
deactivate

git clone https://github.com/opencv/opencv.git
cd opencv
git checkout 3.4
cd ..

git clone https://github.com/opencv/opencv_contrib.git
cd opencv_contrib
git checkout 3.4
cd ..
cd opencv
mkdir build
cd build
cmake -D CMAKE_BUILD_TYPE=RELEASE \
            -D CMAKE_INSTALL_PREFIX=/usr/local \
            -D INSTALL_C_EXAMPLES=ON \
            -D INSTALL_PYTHON_EXAMPLES=ON \
            -D WITH_TBB=ON \
            -D WITH_V4L=ON \
            
		-D OPENCV_PYTHON3_INSTALL_PATH=$cwd/OpenCV-$cvVersion-py3/lib/python3.5/site-packages \
        -D WITH_QT=ON \
        -D WITH_OPENGL=ON \
        -D OPENCV_EXTRA_MODULES_PATH=../../opencv_contrib/modules \
        -D BUILD_EXAMPLES=ON ..
sudo make -j$(nproc)
sudo make install
sudo sed -i 's/CONF_SWAPSIZE=1024/CONF_SWAPSIZE=100/g' /etc/dphys-swapfile
sudo /etc/init.d/dphys-swapfile stop
sudo /etc/init.d/dphys-swapfile start
echo "sudo modprobe bcm2835-v4l2" >> ~/.profile

Or install opencv4.4.0 by https://qengineering.eu/install-opencv-4.4-on-raspberry-pi-4.html
#### 1.1.4 Boost 1.58

wget -O boost_1_58_0.tar.gz https://sourceforge.net/projects/boost/files/boost/1.55.0/boost_1_58_0.tar.gz/download
tar xzvf boost_1_58_0.tar.gz
cd boost_1_58_0/
sudo apt-get update
sudo apt-get install build-essential g++ python-dev autotools-dev libicu-dev build-essential libbz2-dev libboost-all-dev
./bootstrap.sh --prefix=/usr/local
./b2
sudo ./b2 install
sudo ldconfig

#### 1.1.5 QT (my version is 5.7)

sudo apt-get install qt5-default qtcreator

**On your first time to run qtcreator, you need to go to projects/manage kits/build&run/kits/desktop/, and choose the first compiler(GCC c,x86 in /usr/bin) , and click apply and ok.**

#### 1.1.6 V4l2 (current version is 4.4.228)

sudo apt-get install autoconf gettext libtool libjpeg62-dev
git clone git://git.linuxtv.org/v4l-utils.git
cd v4l-utils
autoreconf -vfi
./configure
make
sudo make install

---

## 1.2 Simple install (only need to install QT)

### 1.2.1 INSTALL QT (my version is QT5.7)

sudo apt-get install qt5-default qtcreator

**On your first time to run qtcreator, you need to go to Tools/Options/build&run/kits/desktop/, and choose the first compiler(GCC c,x86 in /usr/bin) , and click apply and ok.**
### 1.2.2 Copy files (Opencv3.1, Boost 1.58, V4l2 )
copy ethan's computer's /usr/local/include to your computer's /usr/local/include
copy ethan's computer's /usr/local/lib to your computer's /usr/local/lib

export LD_LIBRARY_PATH=/usr/local/lib
sudo ldconfig

### 1.2.3 Let v4l2 to be globally used
[Optional] copy v4l2-ctl bin to /usr/bin

cp v4l2-ctl /usr/bin

---

## 1.3 Put V4l2 to your build directory
when you already build,don’t forget to move lib/v4l2 executable into your build directories

sudo mv lib [your project path]/build-XXXXXX

---

### 1.4 autorun:
1.寫好一個你要開機autorun的.sh檔，例如我要在開機時執行 start.sh，我就在桌面上放一個start.sh的檔案
2.在/etc/xdg/autodtart創一個XXX.desktop檔
下面是我的start.sh.desktop檔範例，Exec裡放你執行檔的絕對路徑，Name裡放執行檔的檔名

[Desktop Entry]
Type=Application
Exec=/home/oem/Desktop/start.sh #這裡替換成你的.sh檔路徑
Hidden=false
NoDisplay=false
X-GNOME-Autostart-enabled=true
Name[en_US]=start #需替換成你的.sh檔檔名
Name=start #需替換成你的.sh檔檔名
Comment[en_US]=
Comment=

3.重開機後應該就會自動執行start.sh了

---

## 2.Problems:
### 2.1 Mouse and keyboard issue
When I want to reboot my RPi, sometimes I found I couldn't control my mouse and keyboard.This is because I have overused my cpu.
### 2.2 Select Timeout
Sometimes I write an OpenCV program to control cameras, I met an error message.
> Select Timeout():.....

這是因為我同時用OpenCV開啟三支攝影機，因為樹莓派的cpu相比MiniPC來說較弱，所以有可能在用一個core開啟三支攝影機的時候會跳這個error。

### 2.3 Vector issue
This may be because your libraries' conflicts. You may install OpenCV more than one time. Be careful to handle your OpenCV version. 若你欲在你的電腦裝超過一種版本的OpenCV，那你需將兩個版本的OpenCV lib分開放，當你的project需要其中一種版本的OpenCV的時候，在設置那個版本的路徑，這樣就可以保有兩個版本也不會conflict了。
### 2.4 QT編的執行檔和外面Terminal編的執行檔不同
This may be because QT's default compiler is different with your outside gcc/g++ version. You can choose your compiler in manage kits in qt.
例如: 當外面在編OpenCV的時候要加上 -std=c++11才能編譯，但在QT裡卻不用，另外，之前也發生過用QT的環境編曝光是正常的，但在外面terminal編曝光卻沒有被正確控制的狀況。
### 2.5 Performance
Why we have successfully controlled 3 cameras with 30 fps previously, but why when we ran gui_calibration with C920 with 10 fps, it came out Select Timeout() error? Should we modify our codes with multithread or multicore version?
## 3.Todo List
### 3.1 Upgrade our codes
Fix our codes in calib and make it compatible with Opencv3.4
### 3.2 Minicam 10 fps
Our USB cameras need to have 10 fps options
### 3.3 Migration
以後我們MiniPC的code還會再一直更新，但樹梅派不可能一直追著MiniPC跑，所以我們必須有一個將code移植的步驟，以後每次更新，照這個步驟移植到樹梅派就可以。
1.將 main.cpp裡的if(key!=2)裡的return 0註解掉

#ifndef DEBUG
//return 0
#endif

2.將main.cpp裡的 int relay_device_index那段註解掉

/*int relay_device_index = OS::OSTools::Get8CHRelayDeviceIndex();
if(relay_device_index!=-1)
{
OS::OSTools::Init8CHRelayDevice(relay_device_index);
OS::OSTools::SendData28CHRelay(relay_device_index, 255);
}*///20200727

3. 因為樹梅派插3支攝影機的index分別是024，並不是像MiniPC的是012，而且對樹梅派做"v4l2-ctl --list-device"不知道為甚麼同一個device會跑出很多行不同的結果，因此會造成os/ostools/AvailableCameraDevice裡的parse找不到，我現在的方法是先寫死成024並直接push進camera_list裡。

camera_list.push_back(0);
camera_list.push_back(2);
camera_list.push_back(4);

這方法當然不好，未來我會再尋求更正確的parse方法，或是想辦法讓它做"v4l2-ctl --list-device"的時候可以直接印出一行結果就好。
### 3.4 Multithread or Multicore
Raspberry pi 4B seems hard to run all three cameras in one thread, it will be slow and lagging, so we may need to write another version for multithread or multicore code.