LeRobot Documentation

# LeRobot Documentation Source: LeRobot Official Github: https://github.com/huggingface/lerobot/blob/main/examples/7_get_started_with_real_robot.md ## LeRobot latest example Teleoperation: ``` cd ~/CSL/lerobot_nn conda activate lerobot_nn python -m lerobot.teleoperate --robot.type=koch_follower --robot.port=/dev/ttyUSB_follower --robot.id=my_awesome_follower_arm --robot.cameras="{ front: {type: opencv, index_or_path: /dev/video6, width: 640, height: 480, fps: 30}, top: {type: opencv, index_or_path: /dev/video4, width: 640, height: 480, fps: 30}}" --teleop.type=koch_leader --teleop.port=/dev/ttyUSB_leader --teleop.id=my_awesome_leader_arm --display_data=true ``` ## Try it First! Teleoperation: ``` cd ~/CSL conda activate /home/bruce/anaconda3/envs/lerobot_ethan/ cd lerobot python lerobot/scripts/control_robot.py --robot.type=koch --control.type=teleoperate ``` Visualize dataset: ``` cd ~/CSL conda activate /home/bruce/anaconda3/envs/lerobot/ cd lerobot python lerobot/scripts/visualize_dataset_html.py --repo-id demo/example ``` Inference mode: ``` cd ~/CSL conda activate /home/bruce/anaconda3/envs/lerobot_ethan/ cd lerobot python lerobot/scripts/control_robot.py --robot.type=koch --control.type=record --control.fps=30 --control.single_task="example" --control.repo_id="demo/example" --control.num_episodes=1 --control.push_to_hub=true --control.episode_time_s=50 --control.reset_time_s=10 ``` ![image](https://hackmd.io/_uploads/BJ0AwXWTke.png) Demo: https://drive.google.com/file/d/1VNntEfi1J__0qgROeLTH8pW8_sK7-m50/view?usp=drive_link ## How to train your own imitation learning model? ## Collect data ### Huggingface Token Generation 1. Create Huggingface Token(Write permittion) 2. Run this command with your own huggingface write token ``` huggingface-cli login --token your_token --add-to-git-credential ``` After that, you should see your huggingface user name in this directory: ``` ls -l /home/bruce/.cache/huggingface/lerobot/ bruce HWJ658970 lalalala0620 lerobot StanleyChueh ``` And now, you are ready to train your own Imitation Learning model by LeRobot!! ### Teleoperation & Record dataset Test teleoperation ``` cd ~/CSL conda activate /home/bruce/anaconda3/envs/lerobot_ethan/ cd lerobot python lerobot/scripts/control_robot.py --robot.type=koch --control.type=teleoperate ``` If everything is done, we can start recording the dataset! ``` conda activate /home/bruce/anaconda3/envs/lerobot_ethan/ python lerobot/scripts/control_robot.py --robot.type=koch --control.type=record --control.fps=30 --control.single_task="example" --control.repo_id="demo/example" --control.num_episodes=1 --control.push_to_hub=true --control.episode_time_s=50 --control.reset_time_s=10 ``` After recording the dataset, you can use the following command to visualize the dataset. ``` python lerobot/scripts/visualize_dataset_html.py --repo-id demo/example ``` #### Important Note: 1. press **right key** for saving demo, **left** one for resetting the environment(if the demo fails to finish) 2. I suggest recording at least 50 episodes, with 10 episodes per location. Keep the cameras fixed and maintain consistent grasping behavior throughout the recordings.**REMEMBER** **,** **DATA QUALITY IS CRITICAL IN DATA COLLECTION.** 3. Place Camera to one in front, and one in top-down view 4. The object or target should be seen in **both camera** view as long as possible to keep the better data quality,if the robotic arm block the object in several frames, it will cause poor performance in inference. #### Known Issue 1. It's difficult to use more than two cameras in setting due to the communication speed, the hub is too slow to send camera info if you plug more than one camera. 2. You have to insert THE camera in your PC first(phone camera), and then insert the other one with hub(front camera). ## Train After Collecting dataset, you are ready to train your own imitation learning model! ``` conda activate /home/bruce/anaconda3/envs/lerobot/ python lerobot/scripts/train.py --dataset.repo_id=Your_huggingface_username/task_name --policy.type=act --output_dir=outputs/train/your_task_name --job_name=your_task_name --save_freq=2000 --device=cuda --wandb.enable=true ``` **Important Note:** 1. The default training step is 100K, it takes around 8hours to train. 2. Feel free to test the model performance in training process, you can keep the training terminal open, and create another terminal to test the current model performance. ## Inference After training, it's time to test your own model! ``` conda activate /home/bruce/anaconda3/envs/lerobot_ethan/ python lerobot/scripts/control_robot.py \ --robot.type=koch \ --control.type=record \ --control.fps=30 \ --control.single_task="eval_example" \ --control.repo_id="ethanCSL/eval_example" \ --control.num_episodes=1 \ --control.warmup_time_s=2 \ --control.episode_time_s=300 \ --control.reset_time_s=10 \ --control.push_to_hub=true \ --control.policy.path=outputs/train/example/checkpoints/last/pretrained_model/ ``` **Important Note:** 1. **Every time before you run the inference code, you have to remove the (eval_your_task_name), which is saved in this directory!!!:** ``` /home/bruce/.cache/huggingface/lerobot/your_huggingface_user_name/ ``` eg: ``` ~/.cache/huggingface/lerobot/lalalala0620$ ls eval_act_koch_lego koch_blue_paper_tape koch_pick_place_lego koch_test koch_yellow_paper_tape ``` in this case, delete eval_act_koch_lego, before running the inference mode. ## Experiment Result ### Lego_50 Task: lift lego block Dataset: 50 demos Success rate:10% #### Takeaway: ### Lego_100 Task: lift lego block Dataset: 50 demos Success rate:15~20% #### Takeaway ### Lego_100_class Task: Seperate the lego with yellow and white, and put it into the bin. Dataset: 100 demos Success rate: (70~80% to catch only lego block one by one to its bin) (30~40% if two lego blocks put together) #### Takeaway ### Lego_100_class_test2 Task: Seperate the lego with yellow and white, and put it into the bin. Dataset: 100 demos Success rate:10~15% #### Takeaway 4/1 train 4/2 test 3 檔名：final_test 測資：80筆描述：黑白方塊放入白黃區域直直：20筆橫橫：20筆左斜斜：10筆右斜斜：10筆左右斜斜：20筆先夾白再夾黑成功率:特定位置定擺法60%以上,（位置：左黑右黃並排放中間）,其他位置幾乎無法成功夾取（10%） ### ## Troubleshooting #### Record data If you fail to record dataset, the following things are the things you have to inspect. **1. Permittion Issue** You will have to give access to both camera and robotic arm in the first time they've been plugged in your pc. ``` sudo chmod 666 /dev/video* sudo chmod 666 /dev/ttyUSB* ``` **2.Huggingface user** If you have problem using your own huggingface account while running command, please double check you have give access by the generated write token. (huggingface-cli login --token your_token --add-to-git-credential ) **3.Camera or motor permittion** If you have problem setting the camera or motor, check the permittion or you are using incorrect motor or camera index. Please check the camera and motor index Check robot config: ``` cd lerobot/common/robot_devices/robots/configs.py ``` locate this part: ``` @RobotConfig.register_subclass("koch") @dataclass class KochRobotConfig(ManipulatorRobotConfig): calibration_dir: str = ".cache/calibration/koch" # `max_relative_target` limits the magnitude of the relative positional target vector for safety purposes. # Set this to a positive scalar to have the same value for all motors, or a list that is the same length as # the number of motors in your follower arms. max_relative_target: int | None = None leader_arms: dict[str, MotorsBusConfig] = field( default_factory=lambda: { "main": DynamixelMotorsBusConfig( port="/dev/ttyUSB1", motors={ # name: (index, model) "shoulder_pan": [1, "xl330-m077"], "shoulder_lift": [2, "xl330-m077"], "elbow_flex": [3, "xl330-m077"], "wrist_flex": [4, "xl330-m077"], "wrist_roll": [5, "xl330-m077"], "gripper": [6, "xl330-m077"], }, ), } ) follower_arms: dict[str, MotorsBusConfig] = field( default_factory=lambda: { "main": DynamixelMotorsBusConfig( port="/dev/ttyUSB0", motors={ # name: (index, model) "shoulder_pan": [1, "xl430-w250"], "shoulder_lift": [2, "xl430-w250"], "elbow_flex": [3, "xl330-m288"], "wrist_flex": [4, "xl330-m288"], "wrist_roll": [5, "xl330-m288"], "gripper": [6, "xl330-m288"], }, ), } ) cameras: dict[str, CameraConfig] = field( default_factory=lambda: { "front": OpenCVCameraConfig( camera_index=6, #6 fps=30, width=640, height=480, ), "phone": OpenCVCameraConfig( camera_index=0, fps=30, width=640, height=480, ), } ) ``` Make sure the camera index for "laptop and phone" is compatible to your current setup. Check camera index ``` ls /dev/video* ``` ``` cameras: dict[str, CameraConfig] = field( default_factory=lambda: { "laptop": OpenCVCameraConfig( camera_index=0, fps=30, width=640, height=480, ), "phone": OpenCVCameraConfig( camera_index=1, fps=30, width=640, height=480, ), } ``` Check robotic arm index: ``` ls /dev/ttyUSB* ``` And modify USB port Use this command: ``` python lerobot/scripts/find_motors_bus_port.py ``` ``` leader_arms: dict[str, MotorsBusConfig] = field( default_factory=lambda: { "main": DynamixelMotorsBusConfig( port="/dev/ttyUSB1", motors={ # name: (index, model) "shoulder_pan": [1, "xl330-m077"], "shoulder_lift": [2, "xl330-m077"], "elbow_flex": [3, "xl330-m077"], "wrist_flex": [4, "xl330-m077"], "wrist_roll": [5, "xl330-m077"], "gripper": [6, "xl330-m077"], }, ), } ) follower_arms: dict[str, MotorsBusConfig] = field( default_factory=lambda: { "main": DynamixelMotorsBusConfig( port="/dev/ttyUSB0", motors={ # name: (index, model) "shoulder_pan": [1, "xl430-w250"], "shoulder_lift": [2, "xl430-w250"], "elbow_flex": [3, "xl330-m288"], "wrist_flex": [4, "xl330-m288"], "wrist_roll": [5, "xl330-m288"], "gripper": [6, "xl330-m288"], }, ), } ) ``` ## Train LeRobot with Franka emika panda dataset **1. Navigate to the desired directory** ``` cd ~/.cache/huggingface/lerobot/StanleyChueh ``` **2. Clone from huggingface nyu franka dataset** ``` git clone https://huggingface.co/ethanCSL/three_cube_stack ``` **4.Start Training** On 5090 ``` conda activate lerobot_nn cd lerobot_nn lerobot-train --dataset.repo_id=ethanCSL/pick_n_place_50 --policy.type=act --output_dir=outputs/train/act_pick_n_place_50 --job_name=act_pick_n_place_50 --policy.device=cuda --policy.repo_id=ethanCSL/act_policy --dataset.video_backend=pyav ``` note: tested branch: 1786916a ``` cd lerobot git checkout 1786916a ``` **5.Transfer files between PCs** Make sure these two devices are in the same domain ``` rsync -avzP ~/.cache/huggingface/lerobot/ethanCSL/XXX target_name@target_ip:~/.cache/huggingface/lerobot/ethanCSL/XXX ``` # Franka Lerobot ## Try it now first! Please make sure every terminal is in ROS1 noetic environment, and check is CSL-FET@TT. ``` source /opt/ros/noetic/setup.bash ``` ### Connect to Franka There're two ways to connect to Franka 1. Command line ``` sudo ip addr add 172.16.0.1/24 dev enxc4411e75389a sudo ip addr flush dev enxc4411e75389a sudo ip addr add 172.16.0.1/24 dev enxc4411e75389a sudo ip link set enxc4411e75389a up sudo ufw disable ``` 2. Run the connection code ``` cd ~/franka_record python connect_franka.py ``` ### Launch Franka_ROS ``` cd ~/franka_ws source /opt/ros/noetic/setup.bash source devel/setup.bash roslaunch franka_example_controllers cartesian_impedance_example_controller.launch robot_ip:=172.16.0.2 load_gripper:=true launch_rviz:=false ``` ### Control Robot ``` cd ~/avp_teleoperate_h1/teleop/lerobot_record source /opt/ros/noetic/setup.bash python roboticArm_pose_remote_threshold_ros1_test.py ``` ### Vision Pro Control ``` cd ~/avp_teleoperate_h1/teleop/lerobot_record source /opt/ros/noetic/setup.bash python teleop_arm_pose_threshold_ros1_test.py ``` If you have this error when running the above code: > [Errno 98] error while attempting to bind on address ('0.0.0.0', 8012): address already in use > Check PID > ``` > sudo lsof -i :8012 > ``` > Kill PID > ``` > sudo kill -9 PID > ``` ### LeRobot Record with Franka #### Record ``` cd ~/franka_record/stanley_record source /opt/ros/noetic/setup.bash python record_ros1_test.py --single_task custom_task --repo_id your_huggingface_account/custom_task ``` > Please replace 'custom_task' and 'your_huggingface_account/custom_task' to the actual task and repo_id you want. #### Visualization Push to huggingface ``` cd ~/franka_record/ python push_to_hub.py ``` Note: > If you have trouble pushing to huggingface, please make sure you have the authentication set in your PC, please follow the instruction below: > https://hackmd.io/-TIq0K1NROibtOAh4-sfSQ?both=&stext=1313%3A28%3A0%3A1751419547%3Aeqh_-6 > Now you can do the visualization! ``` conda activate lerobot python lerobot/scripts/visualize_dataset_html.py \ --repo-id your_huggingface_user_name/repo_id ``` ![image](https://hackmd.io/_uploads/Hyym1lgHlx.png) ##### If you don't have your own dataset recorded, please try the following command to see the example: ``` python lerobot/scripts/visualize_dataset_html.py \ --repo-id StanleyChueh/franka_lerobot_red_cube ``` ----------------------------- **Note:** > Please make sure you have dataset in > ``` > ls ~/.cache/huggingface/lerobot/StanleyChueh/franka_lerobot_red_cube > ``` > Otherwise you should re-clone it: > ``` > cd ~/.cache/huggingface/lerobot/StanleyChueh > git clone https://huggingface.co/datasets/StanleyChueh/franka_lerobot_red_cube > ``` ### LeRobot Replay with Franka 1. Launch Franka ROS ``` cd ~/franka_ws source /opt/ros/noetic/setup.bash source devel/setup.bash roslaunch franka_example_controllers cartesian_impedance_example_controller.launch robot_ip:=172.16.0.2 load_gripper:=true launch_rviz:=false ``` 2. Set initial position and switch to impedance control ``` cd ~/avp_teleoperate_h1/teleop/lerobot_record source /opt/ros/noetic/setup.bash python roboticArm_pose_remote_threshold_ros1_test.py ``` > ## Note!!!!!!!!!: > [INFO] [1751621729.042889]: Switched to cartesian impedance controller. > Once this message pops out, you can shut this code down. 3. Replay Episode ``` python ~/franka_record/stanley_record/tools/replay_ros1_v2_quat.py ``` ### LeRobot Training with Franka #### ACT ``` conda activate lerobot cd ~/CSL/lerobot_new python -m lerobot.scripts.train --policy.type=act --dataset.repo_id=user_name/repo_name --output_dir=outputs/train/your_task_name ``` #### SmolVLA ``` python train.py --policy.path=lerobot/smolvla_base --dataset.repo_id=ethanCSL/smolvla_multiblock --batch_size=16 --steps=20000 --output_dir=outputs/train/svla_multiblock --job_name=my_smolvla_training --policy.device=cuda --wandb.enable=false --policy.repo_id=svla_multiblock ``` ### LeRobot Evaluation with Franka #### On Agx Orin Launch socket server ``` conda activate lerobot python lerobot/lerobot/scripts/eval_franka_socket_v2.py ``` #### On control PC(10.100.4.119), On bruce PC(10.100.4.42) ##### Launch Franka_ROS ``` cd ~/franka_ws source /opt/ros/noetic/setup.bash source devel/setup.bash roslaunch franka_example_controllers cartesian_impedance_example_controller.launch robot_ip:=172.16.0.2 load_gripper:=true launch_rviz:=false ``` ##### Control robot ``` cd ~/avp_teleoperate_h1/teleop/lerobot_record source /opt/ros/noetic/setup.bash python roboticArm_pose_remote_threshold_ros1_test.py ``` > ## Note!!!!!!!!!: > [INFO] [1751621729.042889]: Switched to cartesian impedance controller. > Once this message pops out, you can shut this code down. ##### Publish camera topic ``` python ~/franka_record/stanley_record/image_publisher_SA.py ``` ##### Launch socket server & client ``` python evaluation.py --server-ip 10.100.4.42 --target-hz 10 ``` ``` python franka_socket_test_stability_test.py --ckpt-path /home/bruce/CSL/lerobot_nn/src/lerobot/scripts/outputs/train/act_pick_n_place_50/pretrained_model/ --eval-freq 10 ``` ### 📁 Project Structure ```plaintext avp_teleoperate_h1/ ├── act/ ├── assets/ ├── img/ ├── scripts/ ├── teleop/ │ ├── teleop_arm_pose_threshold_ros1_test.py # 🖐️ Hand tracking via Vision Pro │ └── roboticArm_pose_remote_threshold_ros1_test.py # 🤖 Franka Panda control franka_ros/ └── src/ └── franka_ros/ └── franka_example_controllers/ └── launch/ └── cartesian_impedance_example_controller.launch # Launch file for Cartesian impedance control franka_record/ ├── franka_dataset.py # LeRobot-compatible dataset structure └── record_ros1.py # ROS1-based recording script ```

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.