Isaac Sim Pipeline Documentation

[toc] # Isaac Sim Pipeline Documentation --- ## Install Isaac Sim ### On the Server #### 1. Pre-Installation Checks Run the following checks to ensure your environment is ready: - **Test Docker** ```bash docker run --rm hello-world ``` - **Verify GPU Availability** ```bash nvidia-smi ``` - **Check if `nvidia-container-toolkit` is installed** If not installed, the following command will return nothing: ```bash dpkg -l | grep nvidia-container-toolkit ``` #### 2. Pull Isaac Sim Docker Image Find the latest image from [NVIDIA NGC Isaac Sim Catalog](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/isaac-sim): ```bash docker pull nvcr.io/nvidia/isaac-sim:{VERSION} ``` Check available images: ```bash docker images ``` #### 3. Run Isaac Sim in Docker ```bash docker run --name isaac-sim-test --entrypoint bash -it \ --gpus '"device=0"' \ -e "ACCEPT_EULA=Y" \ -e "PRIVACY_CONSENT=Y" \ --rm \ --network=host \ nvcr.io/nvidia/isaac-sim:4.5.0 ``` #### 4. Start Isaac Sim with Native Livestream Launch Isaac Sim headlessly with WebRTC streaming: ```bash ./runheadless.sh # or # new version recommend ./isaac-sim.streaming.sh ``` Once you see `Isaac Sim Full Streaming App is loaded.`, the simulation is running successfully. * Note Use specify GPU ``` CUDA_VISIBLE_DEVICES=0 ./isaac-sim.streaming.sh ``` #### References: - [Isaac Sim Container Installation Docs](https://docs.isaacsim.omniverse.nvidia.com/latest/installation/install_container.html) --- ### On the Local Computer > 💡 This setup assumes the local machine connects to the server via a **local network IP**. #### 1. Install WebRTC Streaming Client Download the latest client from the [official release page](https://docs.isaacsim.omniverse.nvidia.com/latest/installation/download.html#isaac-sim-latest-release). #### 2. Get the Server IP On the server, run: ```bash hostname -I ``` #### 3. Connect to the Server In the WebRTC client, enter the server’s IP address and click **Connect** to start streaming. #### References: - [Live Streaming Docs](https://docs.isaacsim.omniverse.nvidia.com/latest/installation/manual_livestream_clients.html#isaac-sim-short-webrtc-streaming-client) - [WebRTC Streaming Client Release](https://docs.isaacsim.omniverse.nvidia.com/latest/installation/download.html#isaac-sim-latest-release) --- ## Install Isaac Lab ### On the Server (Inside the Isaac Sim Container) #### 1. Clone Isaac Lab Repository Install Git and clone the repo: ```bash apt update && apt install -y git git --version git clone https://github.com/isaac-sim/IsaacLab.git ``` #### 2. Create Symbolic Link to Isaac Sim Navigate to the Isaac Lab directory and create a symbolic link named `_isaac_sim` pointing to the Isaac Sim root: ```bash cd /path/to/IsaacLab ln -s /path/to/isaac-sim _isaac_sim ls -l _isaac_sim ``` #### 3. Install Build Dependencies ```bash apt install cmake build-essential ``` Install Isaac Lab: ```bash cd /path/to/IsaacLab ./isaaclab.sh --install ``` #### 4. Run Isaac Lab with Livestream Enabled Use the WebRTC livestream mode (`--livestream 2`) to connect from a local client. ```bash cd /path/to/IsaacLab ./isaaclab.sh -p path/to/your_script.py \ --headless \ --livestream 2 ``` 📌 Example: Run the base example script: ```bash cd ~/IsaacLab ./isaaclab.sh -p scripts/tutorials/00_sim/launch_app.py \ --headless \ --livestream 2 ``` #### References: - [Installing Isaac Lab Docs](https://isaac-sim.github.io/IsaacLab/main/source/setup/installation/binaries_installation.html#installing-isaac-lab) --- ## Install ROS2 **Prerequisites:** [Installing isaac-sim](https://hackmd.io/jbBMe9ykQ5-ySglpPR_OZg?both#Install-Isaac-Sim) **Location:** inside isaac-sim docker container #### 1. Verify if locale is correct ```bash locale # check for UTF-8 apt-get update && apt-get install locales locale-gen en_US en_US.UTF-8 update-locale LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 export LANG=en_US.UTF-8 locale # verify settings ``` #### 2. Setup sources ```bash apt-get install software-properties-common add-apt-repository universe apt-get update && sudo apt-get install curl -y curl -sSL https://raw.githubusercontent.com/ros/rosdistro/master/ros.key -o /usr/share/keyrings/ros-archive-keyring.gpg echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/ros-archive-keyring.gpg] http://packages.ros.org/ros2/ubuntu $(. /etc/os-release && echo $UBUNTU_CODENAME) main" | tee /etc/apt/sources.list.d/ros2.list > /dev/null ``` #### 3. Install ROS2 packages ```bash apt-get update apt-get upgrade apt-get install -y --allow-downgrades libbrotli1=1.0.9-2build6 apt-get install -y ros-humble-desktop apt-get install -y ros-humble-ros-base apt-get install -y ros-dev-tools apt-get install -y ros-humble-vision-msgs apt-get install -y ros-humble-ackermann-msgs ``` #### 4. Set ROS2 enviroment sourcing the setup script ```bash # Replace ".bash" with your shell if you're not using bash # Possible values are: setup.bash, setup.sh, setup.zsh echo "source /opt/ros/$ROS_DISTRO/setup.bash" >> /root/.bashrc source /opt/ros/humble/setup.bash ``` ## Enabling ROS2 Bridge Extension ```bash pip install roslibpy export ROS_DOMAIN_ID=0 export ROS_IP=127.0.0.1 export ROS_MASTER_URI=http://localhost:11311 export ISAACSIM_ROS2_BRIDGE=1 ``` ## Setting Up Workspaces for ROS2 with isaac-sim #### 1. Clone the ROS2 tutorials workspace ```bash git clone https://github.com/isaac-sim/IsaacSim-ros_workspaces.git ``` #### 2. Move the workspace to /opt ```bash mv IsaacSim-ros_workspaces/humble_ws /opt/ ``` #### 3. Remove redundant files ```bash rm -rf /opt/IsaacSim-ros_workspaces ``` #### 4. Install additional packages ```bash apt-get install -y python3-rosdep python3-rosinstall python3-rosinstall-generator python3-wstool build-essential apt-get install -y python3-colcon-common-extensions ``` #### 5. Create workspaces from original ROS2 downloads ```bash rosdep init && rosdep update || echo "rosdep update failed, check network" source /opt/ros/humble/setup.bash && cd /opt/humble_ws && \ rosdep install -i --from-path src --rosdistro humble -y && \ colcon build ``` #### 6. Set a new ROS2 environment ```bash echo "source /opt/humble_ws/install/local_setup.bash" >> /root/.bashrc source /opt/humble_ws/install/local_setup.bash ``` ## Setup VNC Viewer （x11vnc + xvfb） (Optional) **Optional:** Does not need to be done if GUI is available in the server. since ROS2 uses rviz as a visualization tools. VNC viewer will be used as a viewer that streams the visualization tools from the server to the client. ### On the Server #### 1. Setting VNC ```bash export DEBIAN_FRONTEND=noninteractive export DISPLAY=:1 export TZ=Asia/Taipei ``` #### 2. Install VNC package ```bash apt-get update && apt-get install -y --no-install-recommends \ x11vnc \ xvfb \ supervisor ``` #### 3. Setting VNC Password（Default 0000）& log path ```bash mkdir -p /var/log/x11vnc mkdir -p /root/.vnc x11vnc -storepasswd 0000 /root/.vnc/passwd ``` #### 4. Create Supervisor configuration file `x11vnc.conf` ```bash mkdir -p /etc/supervisor/conf.d/ cat <<EOF > /etc/supervisor/conf.d/x11vnc.conf [program:xvfb] command=/usr/bin/Xvfb :1 -screen 0 1920x1080x24 autostart=true autorestart=true stderr_logfile=/var/log/xvfb.err.log stdout_logfile=/var/log/xvfb.out.log [program:x11vnc] command=/usr/bin/x11vnc -display :1 -forever -usepw -shared autostart=true autorestart=true stderr_logfile=/var/log/x11vnc.err.log stdout_logfile=/var/log/x11vnc.out.log EOF ``` #### 5. Create Launch file `init_services.sh` This script initializes both ROS2 and VNC Services when we run it ```bash #!/bin/bash # Setting ROS2 Env source /opt/ros/humble/setup.bash source /opt/humble_ws/install/local_setup.bash # Launch supervisor to manage x11vnc (run in the background) /usr/bin/supervisord -c /etc/supervisor/supervisord.conf & # Output masseage echo "Custom VNC services initialized in background" ``` #### 6. Launch Supervisor (automatically starts Xvfb and x11vnc) ```bash bash /opt/startup/init_services.sh ``` #### 7. Verify Display ```bash echo $DISPLAY ``` if the output is not 1, then set display to 1 ```bash export DISPLAY=:1 ``` ### On the local computer #### 1. Download VNC viewer client There are other clients, but the link below is **RealVNC** client Download the latest client [here](https://www.realvnc.com/en/connect/download/viewer/windows/) #### 2. Enter Server Address ``` [Server ip]:5900 ``` Enter the password set in step 3 on server setup (the default is `0000`) ## Install Rtab-map (For SLAM) (work in progress) [github-link](https://github.com/introlab/rtabmap_ros/blob/ros2/rtabmap_demos/launch/isaac/isaac_sim_vslam_demo.launch.py) #### 1. Install Additional packages [Isaac ROS package](https://nvidia-isaac-ros.github.io/getting_started/index.html) #### 2. Install rtabmap ```bash apt install ros-$ROS_DISTRO-rtabmap-ros ``` #### 3. Launch Examples in isaac-sim For reference, https://github.com/introlab/rtabmap_ros/blob/ros2/rtabmap_demos/launch/isaac/isaac_sim_vslam_demo.launch.py # 📖 References: - [Isaac Sim Container Installation Docs](https://docs.isaacsim.omniverse.nvidia.com/latest/installation/install_container.html) - [Live Streaming Docs](https://docs.isaacsim.omniverse.nvidia.com/latest/installation/manual_livestream_clients.html#isaac-sim-short-webrtc-streaming-client) - [WebRTC Streaming Client Release](https://docs.isaacsim.omniverse.nvidia.com/latest/installation/download.html#isaac-sim-latest-release) - [ROS2 Installation](https://docs.ros.org/en/humble/Installation/Ubuntu-Install-Debs.html) - [ROS2 Setup](https://docs.omniverse.nvidia.com/isaacsim/latest/installation/install_ros.html#isaac-sim-app-install-native-ros) - [Dockerfile](https://github.com/newbieKD/isaac_ros2_nav2)

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.