## System Setup
1. Log in to [Oscar OpenOnDemand Portal Dashboard](https://ood.ccv.brown.edu/pun/sys/dashboard/).
2. Navigate to > Interactive Apps > Desktop (Advanced).
3. Change `Num Cores` to 1 and `Memory (GB)` to 50. Click **Launch**.
4. Click **Launch Desktop (Advanced)**.
5. Open Terminal.
6. Start an interactive session with 3090-gcondo GPU via the following command:
`interact -n 1 -t 12:00:00 -m 50g -q "3090-gcondo" -X -g 1`
* This requests an interactive session with 1 core and 50 GB of memory (per node) for a period of 12 hours. For more information on the different parameters, check out the following [documentation](https://docs.ccv.brown.edu/oscar/submitting-jobs/interact).
7. Type in the command `nvidia-smi` to check if you now have access to the GPU. Keep in mind that you may have to reach out to your PI / the Oscar team for proper permissions.

## Installation
**Before Installing**

Clone [Ishaan's handpose estimation repository](https://github.com/brown-ivl/hand_pose_estimation).

Load the following modules:

module load cudnn/8.6.0
module load cuda/11.7.1
module load gcc/10.2
module load cmake/3.24.1
module load openssl/3.0.0
module load libarchive/3.6.1
module load curl/7.86.0
module load python/3.9.0
module load leveldb
module load openblas/0.2.19
module load hdf5/1.10.0
module load ffmpeg/4.0.1
module load opencv/3.4.1
module load atlas/3.10.3
module load boost/1.55

Run the following commands to create a new virtual environment:

conda deactivate # deactivate conda in case it is active
python3 -m venv venv
source venv/bin/activate # activate virtual environment

Type the following commands:

export CC=/gpfs/runtime/opt/gcc/10.2/bin/gcc
export CXX=/gpfs/runtime/opt/gcc/10.2/bin/g++
export OPENCV_DIR=/gpfs/runtime/opt/opencv/3.4.1
export PYTHONPATH=":"

The following steps are from [Angela's OpenPose installation guide](https://github.com/brown-ivl/hand_pose_estimation/blob/main/CCVREADME.md). This is the structure she refers to:

hand_pose_estimation
|_ openpose
|_ openpose-reqs
    |_ gflags
    |_ glog
    |_ caffe

Feel free to have your own -- make sure to be consistent!

**Installing Gflags**

git clone https://github.com/gflags/gflags.git
cd gflags
mkdir build
cd build
cmake -DBUILD_SHARED_LIBS=ON \
    -DCMAKE_INSTALL_PREFIX=<hand_pose_estimation dir>/openpose-reqs/gflags/ ..
make install

**Installing Glog**

git clone https://github.com/google/glog.git
cd glog
mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX=<hand_pose_estimation dir>/openpose-reqs/glog/ ..
make install

**Installing Protobuf**

wget https://github.com/protocolbuffers/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.bz2
tar xvf protobuf-2.5.0.tar.bz2
rm protobuf-2.5.0.tar.bz2
cd protobuf-2.5.0
./autogen.sh
./configure -prefix=<hand_pose_estimation dir>/openpose-reqs/protobuf-2.5.0/
make install

**Installing Caffe**

Caffe needs to be installed using Python 2. In a different window, before installing Caffe, load the following modules:

module load cuda/11.7.1
module load gcc/10.2
module load cmake/3.24.1
module load openssl/3.0.0
module load libarchive/3.6.1
module load curl/7.86.0
module load opencv/3.4.1
module load boost/1.55
module load openblas/0.2.19
module load hdf5/1.10.0
module load python/2.7.12

Type the following commands:

export CC=/gpfs/runtime/opt/gcc/10.2/bin/gcc
export CXX=/gpfs/runtime/opt/gcc/10.2/bin/g++
export OpenCV_DIR=/gpfs/runtime/opt/opencv/3.4.1

To install:

git clone https://github.com/BVLC/caffe
cd caffe
mkdir build
cd build
cmake -DUSE_LEVELDB=OFF \
    -DUSE_LMDB=OFF \
    -DBLAS=open \
    -DGLOG_INCLUDE_DIR=<hand_pose_estimation dir>/openpose-reqs/glog/include/ \
    -DGLOG_LIBRARY=<hand_pose_estimation dir>/openpose-reqs/glog/lib64/libglog.so \
    -DGFLAGS_INCLUDE_DIR=<hand_pose_estimation dir>/openpose-reqs/gflags/build/include/ \
    -DGFLAGS_LIBRARY=<hand_pose_estimation dir>/openpose-reqs/gflags/build/lib/libgflags.so \
    -DPROTOBUF_INCLUDE_DIR=<hand_pose_estimation dir>/openpose-reqs/protobuf-2.5.0/include \
    -DPROTOBUF_LIBRARY=<hand_pose_estimation dir>/openpose-reqs/protobuf-2.5.0/lib/libprotobuf.so \
    -DPROTOBUF_PROTOC_EXECUTABLE=<hand_pose_estimation dir>/openpose-reqs/protobuf-2.5.0/bin/protoc \
    -DOpenBLAS_INCLUDE_DIR=/gpfs/runtime/opt/openblas/0.2.19/include \
    -DOpenBLAS_LIB=/gpfs/runtime/opt/openblas/0.2.19/lib/libopenblas.so \
    -DCMAKE_INSTALL_PREFIX=<hand_pose_estimation dir>/openpose-reqs/caffe ..
make install

**File Modifications**

Before installing Openpose, the current code needs to be modified to allow for 3D keypoint extraction via Python. Copy this file [openpose_python.cpp](https://github.com/ishaanshah/openpose-handpose-3d/blob/main/openpose_python.cpp) into `<OPENPOSE_DIR>/python/openpose`. There will be an existing file that you will replace. There are a few ways to do this; here is one of them:

1. Log in to the [Oscar OpenOnDemand Portal Dashboard](https://ood.ccv.brown.edu/pun/sys/dashboard).
2. Navigate to Home Directory.
3. Navigate to `<OPENPOSE_DIR>/python/openpose`. If you followed Angela's structure above, this would be `hand_pose_estimation/openpose/python/openpose`.
4. Click the dropdown menu for the `openpose_python.cpp` file and click Edit.
5. Directly copy the contents of [the file above]((https://github.com/ishaanshah/openpose-handpose-3d/blob/main/openpose_python.cpp)), and **press Save (blue button on top left)**.

This isn't the only file that needs to be changed. While we're at it, navigate to `<OPENPOSE_DIR>/src/openpose/3d/poseTriangulationPrivate.cpp` and click edit. Now, on line 163, change 8 to 100. **Press Save**. This modification allows us to triangulate with more than 8 views.

**Installing OpenPose**

git clone https://github.com/CMU-Perceptual-Computing-Lab/openpose
cd openpose
git submodule update --init --recursive --remote
mkdir build
cd build
cmake -DBUILD_PYTHON=ON \
    -DGPU_MODE=CUDA \
    -DUSE_CUDNN=TRUE \
    -DCMAKE_INSTALL_PREFIX=<hand_pose_estimation dir>/openpose/ \
    -DProtobuf_INCLUDE_DIR=<hand_pose_estimation dir>/openpose-reqs/protobuf-2.5.0/include/ \
    -DProtobuf_LIBRARIES=<hand_pose_estimation dir>/openpose-reqs/protobuf-2.5.0/lib/ \
    -DCUDNN_INCLUDE=/gpfs/runtime/opt/cudnn/8.6.0/include/ \
    -DCUDNN_LIBRARY=/gpfs/runtime/opt/cudnn/8.6.0/lib/libcudnn.so \
    -DGLOG_INCLUDE_DIR=<hand_pose_estimation dir>/openpose-reqs/glog/include/ \
    -DGLOG_LIBRARY=<hand_pose_estimation dir>/openpose-reqs/glog/lib64/libglog.so \
    -DGFLAGS_INCLUDE_DIR=<hand_pose_estimation dir>/openpose-reqs/gflags/include/ \
    -DGFLAGS_LIBRARY=<hand_pose_estimation dir>/openpose-reqs/gflags/lib/libgflags.so \
    -DCaffe_INCLUDE_DIRS=<hand_pose_estimation dir>/openpose-reqs/caffe/include/ \
    -DCaffe_LIBS=<hand_pose_estimation dir>/openpose-reqs/caffe/lib64/libcaffe.so \
    -DBUILD_CAFFE=OFF ..
make install

# these commands will make sure OpenPose runs without error in the hand pose estimation pipeline
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:<hand_pose_estimation dir>/openpose-reqs/gflags/lib
export LD_PRELOAD=/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64/libmkl_core.so:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64/libmkl_sequential.so

## Using OpenPose

To make OpenPose available to your virtual environment, you need to add it to your `site-packages` directory. To do so, run the following commands:

export OPENPOSE_DIR=<OpenPose Path>
pushd <venv_path>/lib/<python_version>/site-packages/
ln -s $OPENPOSE_DIR/build/python/openpose .
popd

**Installing Dependencies to Virtual Environment**

Install PyTorch in your virtual environment by running this command:

python3 -m pip install torch==2.0.0+cu117 torchvision==0.15.1+cu117 torchaudio==2.0.1 --index-url https://download.pytorch.org/whl/cu117

* Note: Make sure, when you run this command, that the first line **does not** say "Defaulting to user installation because normal site-packages is not writeable." If you get this error, try restarting your virtual environment. For any unrelated errors, try reloading all of the modules from above.

With PyTorch successfully installed, you can now install the required dependencies for the virtual environment. Navigate to your `hand_pose_estimation` directory and run `pip install -r requirements.txt`.

Now, run the following commands:

pip uninstall ffmpeg
pip install ffmpeg-python

**Running OpenPose**

Navigate to `hand_pose_estimation/preprocess.sh` and edit the relevant paths based on the comments.

Navigate to `hand_pose_estimation/scripts/get_3d_keypoints.py` and change `from openpose import pyopenpose as op` to `import pyopenpose as op`.

The file tree expects the following structure:

hand_pose_estimation
└── dataset
    └── [name of your sequence/data] 
        ├── image
        |   ├── synced
        |   │   ├── brics-sbc-001_cam0
        │   |   ├── brics-sbc-002_cam0
        │   |   ├── brics-sbc-003_cam0
        │   |   ├── brics-sbc-003_cam1
        └── range.txt

In `hand_pose_estimation`, make a directory for your data.

I'm doing it for a sequence related to making tea, so I'm calling mine `making_tea`. Within this directory, make a directory called `image`. Within `image`, make a `synced` directory as well as a text file called `range.txt`.

Run the following command:

sh preprocess.sh -r [path to hand_pose_estimation/dataset] -s [name of your sequence, i.e. making_tea]

## Potential Errors
Before anything, try:

1. Reloading the modules.
2. Deactivating and reactivating your virtual environment.
3. Rerunning the `export` commands from above. Double check that your file paths are correct.

### Installing Openpose
* Issues with Protobuf
* Make sure that you **haven't** installed requirements (`pip install -r requirements.txt`) before installing OpenPose.
* The order should be:
1. Make and activate virtual environment.
2. Install OpenPose dependencies and OpenPose.
3. Install virtual environment requirements (PyTorch first, and then `requirements.txt`).
* If you've already installed the virtual environment requirements before the OpenPose dependencies, you'll have to deactivate, delete the virtual environment (`rm venv -rf`), and then remake it.

### Running Openpose
* `Attribute Error: 'openpose.pyopenpose.WrapperPython' object has no attribute 'detectHandKeypoints3D'`
* Check the `openpose_python.cpp` file in your `<OPENPOSE_DIR>/python/openpose` directory. Ctrl + F and see if there's a `detectHandKeypoints3D` function. If not, it means that this file was not successfully replaced.
* Try to replace it again according to the instructions above. Don't forget to save!!!
* I ran into an issue where the file always reverted back to its original contents as long as it had the name `openpose_python.cpp`. If this is happening to you, delete the original file, and then rename the new one to `openpose_python2.cpp` before adding it to the directory. If you do this, you also have to update the `CMakeLists.txt` file within the same directory: replace every occurrence of "openpose_python.cpp" with "openpose_python2.cpp".
* If you've run into this error, it's likely you'll have the same problem when trying to modify the file `poseTriangulationPrivate.cpp`. Double check that the update has been made -- if not, you may have to similarly rename the file.
* If you run into this error, you will need to rebuild OpenPose and your virtual environment.

* `ModuleNotFoundError: No module named 'ffmpeg'` or
`[Errno 2] No such file or directory: 'ffprobe': 'ffprobe'`
* If you've already uninstalled `ffmpeg` and installed `ffmpeg-python`, try reloading the module:

        module unload ffmpeg
        module load ffmpeg/4.0.1