Isaac Sim - HackMD

<span style="font-size: 32px;">**Isaac Sim**</span> If there are any errors in the article, please correct me, thank you. --- [toc] # Install Isaac Sim ## Docker runs Isaac Sim connect to Omniverse steaming client **( Headless remote server )** ### In server 1. docker test ``` docker run --rm hello-world ``` 2. check GPU ``` nvidia-smi ``` * check Nvidia-conatainer-toolkit, If it is not installed, nothing will be displayed. ``` dpkg -l | grep nvidia-container-toolkit ``` 3. Pull Container [Images source](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/isaac-sim) ``` docker pull nvcr.io/nvidia/isaac-sim:{Version} ``` ``` docker images ``` 4. Run docker ``` docker run --name isaac-sim-test --entrypoint bash -it \ --gpus '"device=0"' \ -e "ACCEPT_EULA=Y" \ --rm \ --network=host \ -e "PRIVACY_CONSENT=Y" \ nvcr.io/nvidia/isaac-sim:4.5.0 ``` ``` docker run --name isaac-sim-test --entrypoint bash -it \ --gpus '"device=0"' \ -e "ACCEPT_EULA=Y" \ --rm \ --network=host \ -e "PRIVACY_CONSENT=Y" \ -v /path/to/folder/isaac-sim/cache/kit:/isaac-sim/kit/cache:rw \ -v /path/to/folder/isaac-sim/cache/ov:/root/.cache/ov:rw \ -v /path/to/folder/isaac-sim/cache/pip:/root/.cache/pip:rw \ -v /path/to/folder/isaac-sim/cache/glcache:/root/.cache/nvidia/GLCache:rw \ -v /path/to/folder/isaac-sim/cache/computecache:/root/.nv/ComputeCache:rw \ -v /path/to/folder/isaac-sim/logs:/root/.nvidia-omniverse/logs:rw \ -v /path/to/folder/isaac-sim/data:/root/.local/share/ov/data:rw \ -v /path/to/folder/isaac-sim/documents:/root/Documents:rw \ nvcr.io/nvidia/isaac-sim:4.5.0 ``` ``` docker run --name isaac-sim-test --entrypoint bash -it --gpus '"device=0"' -e "ACCEPT_EULA=Y" --rm --network=host \ -e "PRIVACY_CONSENT=Y" \ -v /m****/Pl***/{User_name}/{folder_name}/isaac-sim/cache/kit:/isaac-sim/kit/cache:rw \ -v /m****/Pl***/{User_name}/{folder_name}/isaac-sim/cache/ov:/root/.cache/ov:rw \ -v /m****/Pl***/{User_name}/{folder_name}/isaac-sim/cache/pip:/root/.cache/pip:rw \ -v /m****/Pl***/{User_name}/{folder_name}/isaac-sim/cache/glcache:/root/.cache/nvidia/GLCache:rw \ -v /m****/Pl***/{User_name}/{folder_name}/isaac-sim/cache/computecache:/root/.nv/ComputeCache:rw \ -v /m****/Pl***/{User_name}/{folder_name}/isaac-sim/logs:/root/.nvidia-omniverse/logs:rw \ -v /m****/Pl***/{User_name}/{folder_name}/isaac-sim/data:/root/.local/share/ov/data:rw \ -v /m****/Pl***/{User_name}/{folder_name}/isaac-sim/documents:/root/Documents:rw \ nvcr.io/nvidia/isaac-sim:4.5.0 ``` `/m****/Pl***/{User_name}/{folder_name}/isaac-sim/`is the location of the local folder, so that the data after closing docker will also be stored. Please see the [official website](https://docs.omniverse.nvidia.com/isaacsim/latest/installation/install_container.html#isaac-sim-setup-remote-headless-container) for details. 5. Start Isaac Sim with native livestream mode ``` # isaac sim 4.2.0 ./runheadless.native.sh ./runheadless.native.sh apps/omni.create.kit \ --enable omni.services.streaming.manager \ --enable omni.kit.livestream.native \ --no-window \ --allow-root \ --renderer RTX \ --/app/window/dpiScaleOverride=1.0 \ -v ./isaac-sim.headless.native.sh apps/omni.create.kit \ --enable omni.services.streaming.manager \ --enable omni.kit.livestream.native \ --no-window \ --allow-root \ --renderer RTX \ --/app/window/dpiScaleOverride=1.5 \ # isaac sim 4.5.0 # For omniverse streaming server [DEPRECATED] ./isaac-sim.old_streaming.sh apps/omni.create.kit \ --enable omni.kit.livestream.native \ --no-window \ --allow-root \ --renderer RTX \ --/app/window/dpiScaleOverride=1.5 \ # For Isaac Sim WebRTC Streaming Client ./runheadless.sh ``` For isaac sim 4.2.0 : When you see the message `Isaac Sim Headless Native App is loaded.`, it means that Isaac Sim has successfully started. Using different `.sh` file to open Isaac Sim will obtain different results. The results here only correspond to using `./runheadless.native.sh`. Please see the [official website](https://docs.omniverse.nvidia.com/isaacsim/latest/installation/manual_livestream_clients.html) for details. For Isaac sim 4.5.0 : When you see the message `Isaac Sim Full Streaming App is loaded.`, it means that Isaac Sim has successfully started. Refer : * [live streaming](https://docs.isaacsim.omniverse.nvidia.com/latest/installation/manual_livestream_clients.html#isaac-sim-short-webrtc-streaming-client) * [Isaac Sim WebRTC Streaming Client Latest Release](https://docs.isaacsim.omniverse.nvidia.com/latest/installation/download.html#isaac-sim-latest-release) ### In local * **This connection is established using a local network IP. If anyone knows how to connect using a public IP or other methods, your insights would be greatly appreciated.** 1. Install and launch the Omniverse Streaming Client from Omniverse Launcher **(deprecated)**, or use the latest Isaac Sim WebRTC Streaming Client. * [Isaac Sim WebRTC Streaming Client Latest Release](https://docs.isaacsim.omniverse.nvidia.com/latest/installation/download.html#isaac-sim-latest-release) 2. Find server IP ``` hostname -I ``` Output: `192.168.X.XX 172.17.X.XX ` 3. Enter the IP address of the machine or instance running the Isaac Sim container and click on the Connect button to begin live streaming. In this example enter the IP as `192.168.X.XX`. ### Use python on Isaac Sim (Optional) **On the local side, need to install VSCode along with `Dev Containers`, `Remote - SSH`, `Remote - SSH: Editing Configuration Files`, and `Remote Explorer`. On the server side, `Isaac Sim VS Code Edition` is required. This allows you to edit Python code on the local side and execute it on the server side, with the results displayed in the GUI interface.** # Install Isaac lab * There seems to be other ways to use docker to start isaac lab, but I haven't read it in detail. refer to:[Docker Guide](https://isaac-sim.github.io/IsaacLab/main/source/deployment/docker.html#directory-organization) ## If use docker to start Isaac sim **( Headless remote server )** ### In server * **The following is run in the started Isaac sim docker** 1. Clone the Isaac Lab ``` apt update && apt install -y git git --version git clone https://github.com/isaac-sim/IsaacLab.git ``` 2. Creating the Isaac Sim Symbolic Link Set up a symbolic link between the installed Isaac Sim root folder and _isaac_sim in the Isaac Lab directory. This makes it convenient to index the python modules and look for extensions shipped with Isaac Sim. `cd /path/to/isaac-sim/IsaacLab` ``` ln -s path_to_isaac_sim _isaac_sim # For example: ln -s /isaac-sim _isaac_sim ls -l _isaac_sim ``` 3. Installation ``` apt install cmake build-essential ``` `cd /path/to/isaac-sim/IsaacLab` ``` ./isaaclab.sh --install # or "./isaaclab.sh -i" ``` 4. Start Isaac Lab with native livestream mode > In the following example, `--livestream 1` is used for Omniverse Streaming Client **(deprecated)**, while `--livestream 2` is used for Isaac Sim WebRTC Streaming Client. > Written on 2025-02-15. `cd /path/to/isaac-sim/IsaacLab` ``` ./isaaclab.sh -p path/to/{aim.py_file} \ --headless \ --livestream 1 \ --kit_args="--/app/window/dpiScaleOverride=1.5" \ ``` **The paths in the following examples may change with different versions and are provided for reference only** Example: File path: `~/IsaacLab/scripts/tutorials/00_sim/create_empty.py` `cd /path/to/isaac-sim/IsaacLab` ``` # isaac sim 4.5.0 ./isaaclab.sh -p scripts/tutorials/00_sim/launch_app.py --headless \ --livestream 1 \ --kit_args="--/app/window/dpiScaleOverride=1.5" \ ``` Train example: task list path: ~~`~/IsaacLab/source/extensions/omni.isaac.lab_tasks/omni/isaac/lab_tasks/direct/cartpole/__init__.py`~~ New path: `~/IsaacLab/source/isaaclab_tasks/isaaclab_tasks/direct/cartpole/__init__.py` `cd /path/to/isaac-sim/IsaacLab` ``` # isaac sim 4.2.0 / isaac lab 1.0 ./isaaclab.sh -p source/standalone/workflows/rl_games/train.py \ --task=Isaac-Cartpole-Direct-v0 \ --headless \ --livestream 1 \ --kit_args="--/app/window/dpiScaleOverride=1.5" \ # new path ./isaaclab.sh -p scripts/reinforcement_learning/rl_games/train.py \ --task=Isaac-Cartpole-Direct-v0 \ --headless \ --livestream 1 \ --kit_args="--/app/window/dpiScaleOverride=1.5" \ ``` Please see the [official website(RL train)](https://isaac-sim.github.io/IsaacLab/main/source/tutorials/03_envs/run_rl_training.html) for details. Spot dog path: `/IsaacLab/source/isaaclab_tasks/isaaclab_tasks/manager_based/locomotion/velocity/config/spot/__init__.py` > ./isaaclab.sh -p scripts/reinforcement_learning/rsl_rl/train.py \ --task Isaac-Velocity-Flat-Spot-v0 \ --num_envs 4096 \ --headless \ --livestream 1 \ --kit_args="--/app/window/dpiScaleOverride=1.5" \ > ./isaaclab.sh -p scripts/reinforcement_learning/rsl_rl/play.py \ --task Isaac-Velocity-Flat-Spot-v0 \ --num_envs 64 \ --headless \ --livestream 1 \ --kit_args="--/app/window/dpiScaleOverride=1.5" \ ### In local * Same as before # install ROS2 * Written on 2025/03/24 ## ROS2 Container Installation and Cross-Container Communication with Isaac Sim This note documents the steps to create a ROS2 container, download and build the Isaac Sim official ROS workspace, and mount the ROS2 container into the Isaac Sim container to enable cross-container communication. --- ### 1. Creating the ROS2 Container on the Server #### 1.1 Add the ROS package source and install rocker ```bash sudo apt-key adv --keyserver 'hkp://keyserver.ubuntu.com:80' --recv-key C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654 sudo sh -c 'echo "deb http://packages.ros.org/ros/ubuntu $(lsb_release -sc) main" > /etc/apt/sources.list.d/ros-latest.list' sudo apt-get update sudo apt-get install python3-rocker sudo apt autoremove ``` #### 1.2 Start the ROS2 Container (We use the OSRF provided image: `osrf/ros:humble-desktop-full-jammy`) ```bash rocker --nvidia --x11 --privileged --network host --name ROS2 --nocleanup \ osrf/ros:humble-desktop-full-jammy ``` > **Note:** To keep the container running long-term, you can use tmux: > ```bash > tmux new-session -d 'rocker --nvidia --x11 --privileged --network host --name <container name> osrf/ros:humble-desktop-full-jammy' > ``` #### 1.3 Exit the ROS2 Container ```bash exit ``` --- ### 2. Downloading and Building the Isaac Sim ROS Workspace #### 2.1 Download the official ROS workspace from Isaac Sim ```bash cd /home/user git clone https://github.com/isaac-sim/IsaacSim-ros_workspaces.git ``` #### 2.2 Copy the workspace into the running ROS2 container ```bash docker cp /home/user/IsaacSim-ros_workspaces/humble_ws ROS2:/root/humble_ws ``` #### 2.3 Enter the ROS2 Container <container name> = ROS2 ```bash docker exec -it <container name> bash ``` #### 2.4 Build the ROS2 Workspace ```bash source /opt/ros/humble/setup.sh cd /root/humble_ws apt-get update rosdep install --from-paths src --ignore-src --rosdistro=humble -y source /opt/ros/humble/setup.sh colcon build source install/local_setup.bash ``` #### 2.5 Set ROS2-related environment variables and test ```bash export ROS_DOMAIN_ID=0 export FASTRTPS_DEFAULT_PROFILES_FILE=~/.ros/fastdds.xml source /opt/ros/humble/setup.sh # Verify functionality ros2 topic list ros2 node list ``` --- ### 3. Creating a New ROS2 Image and Mounting as a Volume #### 3.1 Copy the ROS2 installation folder from the ROS2 container to the server ```bash docker cp ROS2:/opt/ros/humble /home/user/docker/ROS2/humble ``` #### 3.2 Commit the ROS2 Container to a New Image ```bash docker commit ROS2 new_ros2_image:3.24 ``` #### 3.3 Stop and Remove the Original ROS2 Container ```bash docker stop ROS2 docker rm ROS2 ``` #### 3.4 Start a New ROS2 Container with Volume Mounting ```bash rocker --nvidia --x11 --privileged --network host --name ROS2 \ --nocleanup \ --volume /home/user/docker/ROS2/humble:/opt/ros/humble \ -- new_ros2_image:3.24 ``` --- ### 4. Starting the Isaac Sim Container and Mounting the ROS2/humble Folder Run the following command on the server to start the Isaac Sim container and mount the ROS2/humble folder: ```bash docker run --name isaac-sim-robotic --entrypoint bash -it --gpus '"device=1"' -e "ACCEPT_EULA=Y" --network=host \ -e "PRIVACY_CONSENT=Y" \ -v /home/user/docker/isaac-sim/cache/kit:/isaac-sim/kit/cache:rw \ -v /home/user/docker/isaac-sim/cache/ov:/root/.cache/ov:rw \ -v /home/user/docker/isaac-sim/cache/pip:/root/.cache/pip:rw \ -v /home/user/docker/isaac-sim/cache/glcache:/root/.cache/nvidia/GLCache:rw \ -v /home/user/docker/isaac-sim/cache/computecache:/root/.nv/ComputeCache:rw \ -v /home/user/docker/isaac-sim/logs:/root/.nvidia-omniverse/logs:rw \ -v /home/user/docker/isaac-sim/data:/root/.local/share/ov/data:rw \ -v /home/user/docker/isaac-sim/documents:/root/Documents:rw \ -v /home/user/docker/IsaacLab:/IsaacLab:rw \ -v /home/user/docker/ROS2/humble:/opt/ros/humble:rw \ --env ROS_DISTRO=humble \ --env AMENT_PREFIX_PATH=/opt/ros/humble \ --env RMW_IMPLEMENTATION=rmw_fastrtps_cpp \ --env LD_LIBRARY_PATH=/isaac-sim/exts/isaacsim.ros2.bridge/humble/lib:$LD_LIBRARY_PATH \ isaac-sim-isaac-lab-robotic:3.16 ``` --- ### 5. Setting Up the ROS2 Environment Inside the Isaac Sim Container Once inside the Isaac Sim container, execute the following steps: 1. **Set PATH and Create a Symbolic Link for Python3** ```bash export PATH=/isaac-sim/kit/python/bin:$PATH ln -s /isaac-sim/kit/python/bin/python3 /usr/bin/python3 ``` 2. **Set ROS2 and ROS2 Bridge Related Environment Variables** ```bash export ROS_DOMAIN_ID=0 export RMW_IMPLEMENTATION=rmw_fastrtps_cpp export LD_LIBRARY_PATH=/opt/ros/humble/lib:$LD_LIBRARY_PATH export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/isaac-sim/exts/omni.isaac.ros2.bridge/humble/lib export AMENT_PREFIX_PATH=/opt/ros/humble export ROS_DISTRO=humble ``` 3. **Source the ROS2 Environment** ```bash source /opt/ros/humble/setup.bash ``` 4. **(Option)Install Missing tinyxml2 Packages** ```bash apt-get update apt-get install -y libtinyxml2-9 libtinyxml2-dev # Check installation: ls -l /usr/lib/x86_64-linux-gnu/libtinyxml2.so.9* ``` --- ### 6. Final Checks Inside the Isaac Sim container, verify that the ROS2 Bridge extension starts without libtinyxml2 errors: ```bash cd /isaac-sim ./runheadless.sh ``` output: ``` [13.071s] [ext: isaacsim.ros2.bridge-4.1.15] startup [13.226s] Attempting to load system rclpy [13.245s] rclpy loaded ``` --- ### Note - Ensure both containers are using `--network host` or have a properly configured Docker network to allow cross-container communication. - Adjust paths, container names, and image tags as needed based on your environment. - The paths used above (`/home/user` and `/home/user/docker`) are for public documentation purposes; you can modify them as appropriate for your setup. This completes the process for installing the ROS2 container and enabling cross-container communication with Isaac Sim.

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.