# OpenVLN Documentation

###### tags: `Notes`

- Auto-generated Table of Content
[ToC]

___

ssh [e-mail]
___

Create a conda env:

conda create -n openvln python=3.8 cmake=3.14.0
conda activate openvln

Load newer version of GCC to allow correct use of numba package. (we need gcc at least 11.2.0); as well as corresponding NVCC.

module load apps/cmake/3.25.1/gcc-11.2.0
module load libs/nvidia-cuda/12.1.1/bin

## 1. Install Prismatic VLMs

Following instruction in following link to install VLM codebase first:

https://github.com/TRI-ML/prismatic-vlms

or following:

git clone https://github.com/TRI-ML/prismatic-vlms
cd prismatic-vlms
pip install -e .

# Training additionally requires Flash-Attention 2 (https://github.com/Dao-AILab/flash-attention)
pip install packaging ninja

# Verify Ninja --> should return exit code "0"
ninja --version; echo $?

# Install Flash Attention 2 
#   =>> If you run into difficulty, try `pip cache remove flash_attn` first
pip install flash-attn --no-build-isolation

## 2. Installing Habitat Simulator and Lab Tools

Installing Habitat sim and lab version 0.2.2.

### Habitat-lab

Install habitat-lab v0.2.2:

git clone https://github.com/facebookresearch/habitat-lab.git
cd habitat-lab
git checkout v0.2.2


pip install -r requirements.txt
python setup.py develop --all # install habitat and habitat_baselines

# i know we need this version!! (after 100000000 attempts..)
pip install numpy==1.21.6

If the installation reports error with moviepy version, try the following and run `python setup.py develop --all` again:

pip uninstall moviepy
pip install moviepy

### Habitat-sim

Set system varible to enable it if needed (remember to change the path to the actual gcc path in your env):

export PATH=/opt/gridware/depots/e2b91392/el7/pkg/compilers/gcc/11.2.0/bin:$PATH
export LD_LIBRARY_PATH=/opt/gridware/depots/e2b91392/el7/pkg/compilers/gcc/11.2.0/lib:/opt/gridware/depots/e2b91392/el7/pkg/compilers/gcc/11.2.0/lib64:$LD_LIBRARY_PATH

Install habitat-sim v0.2.2:

conda install habitat-sim==0.2.2 withbullet headless -c conda-forge -c aihabitat

### Test your install

Downloading test scenes and objects:

python -m habitat_sim.utils.datasets_download --uids habitat_test_scenes --data-path /path/to/data/
python -m habitat_sim.utils.datasets_download --uids habitat_example_objects --data-path /path/to/data/

Testing Habitat-Lab：

cd habitat-lab
python examples/example.py

## 3. Install OpenVLN

### Install

Clone repo:

git clone https://github.com/HuskyKingdom/diffuser_navigator.git

cd diffuser_navigator

pip install -r requirements.txt

### Prepare Dataset

The data folder (after downloading them all) is expected to have the following form:

data/
├── checkpoints/eval_copy/ckpt.100.pth
├── datasets/
├── ddppo-models/
├── res/
├── scene_datasets/
├── tensorboard_dirs/
├── trajectories_dirs/diffuser
└── connectivity_graphs.pkl

[datasets](https://drive.google.com/file/d/1CxLHnd6qZagVe0AI6BYVMiyfNb1E0AzN/view?usp=sharing)

[ddppo-models](https://drive.google.com/file/d/19FWa0hv0xqSEGiN-8f2hmaU5IL1UcGS0/view?usp=sharing)

[scene_datasets](https://drive.google.com/file/d/1qPZ-10N2GeibBHbGzW-DRGfhf6vKDB3k/view?usp=sharing)

## 4. Configs

Login to your wandb account:

wandb login

We need access to the gated opensource pre-trained LLMs. Require the access [https://huggingface.co/meta-llama/Llama-2-7b-chat-hf](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf).

Then create a new file `.hf_token` for the code to recon. your identity:

# create & insert your token into the file 
vim .hf_token

## 5. Collecting Datas

For teacher-forcing action alignment:

sbatch openvln_phase1_datacollecting.sbatch

For CoT training phase:

sbatch openvln_phase2_datacollecting.sbatch

### Q&A

#### Q

`Resolution too deep` Erro while installing Prismatic VLM.

#### A

Manully install torch huggingface-hub and draccus, then try again.

pip install torch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 --index-url https://download.pytorch.org/whl/cu121
pip install huggingface-hub==0.27.1
pip install draccus==0.9.3
pip install accelerate==1.0.1
pip install transformers==4.46.3
pip install tokenizers==0.20.3
pip install -e .

#### Q

 File "/users/sgyson10/volatile/miniconda3/envs/openvln/lib/python3.8/site-packages/gym/spaces/discrete.py", line 21, in __init__
    assert n > 0, "n (counts) have to be positive"

#### A

This happens to VLNCE original repo too, it is likely an issue with gym version, change the corresponding value in code (`habitat-lab/habitat/tasks/vln/vln.py` line 60) can avoid it：

self.observation_space = spaces.Discrete(0) 
-> 
self.observation_space = spaces.Discrete(4)

export UCX_NET_DEVICES=mlx5_0:1
export UCX_IB_GPU_DIRECT_RDMA=1

https://butui.me/posts/notes-on-nccl-as-pytorch-ddp-backend/