# LeRobot Documentation
Source: LeRobot Official Github:
https://github.com/huggingface/lerobot/blob/main/examples/7_get_started_with_real_robot.md

# Now all the things have moved to https://github.com/StanleyChueh/lerobot.git, Main branch is for dataset format v2.1,and dev branch is for dataset v3.0.
## LeRobot latest example
Teleoperation:

cd ~/CSL/lerobot_nn
conda activate lerobot_nn
python -m lerobot.teleoperate     --robot.type=koch_follower     --robot.port=/dev/ttyUSB_follower     --robot.id=my_awesome_follower_arm     --robot.cameras="{ front: {type: opencv, index_or_path: /dev/video6, width: 640, height: 480, fps: 30}, top: {type: opencv, index_or_path: /dev/video4, width: 640, height: 480, fps: 30}}"     --teleop.type=koch_leader     --teleop.port=/dev/ttyUSB_leader     --teleop.id=my_awesome_leader_arm     --display_data=true

## Try it First!
Teleoperation:

cd ~/CSL
conda activate /home/bruce/anaconda3/envs/lerobot_ethan/
cd lerobot
python lerobot/scripts/control_robot.py   --robot.type=koch   --control.type=teleoperate

Visualize dataset:

cd ~/CSL
conda activate /home/bruce/anaconda3/envs/lerobot/
cd lerobot
python lerobot/scripts/visualize_dataset_html.py   --repo-id demo/example

Inference mode:

cd ~/CSL
conda activate /home/bruce/anaconda3/envs/lerobot_ethan/
cd lerobot
python lerobot/scripts/control_robot.py     --robot.type=koch     --control.type=record     --control.fps=30     --control.single_task="example"     --control.repo_id="demo/example"     --control.num_episodes=1     --control.push_to_hub=true     --control.episode_time_s=50     --control.reset_time_s=10  


Demo:
https://drive.google.com/file/d/1VNntEfi1J__0qgROeLTH8pW8_sK7-m50/view?usp=drive_link

## How to train your own imitation learning model?
## Collect data

### Huggingface Token Generation

1. Create Huggingface Token(Write permittion)
2. Run this command with your own huggingface write token

huggingface-cli login --token your_token --add-to-git-credential 

After that, you should see your huggingface user name in this directory:

ls -l /home/bruce/.cache/huggingface/lerobot/
bruce  HWJ658970  lalalala0620  lerobot  StanleyChueh

And now, you are ready to train your own Imitation Learning model by LeRobot!!
### Teleoperation & Record dataset

Test teleoperation

cd ~/CSL
conda activate /home/bruce/anaconda3/envs/lerobot_ethan/
cd lerobot
python lerobot/scripts/control_robot.py   --robot.type=koch   --control.type=teleoperate

If everything is done, we can start recording the dataset!

conda activate /home/bruce/anaconda3/envs/lerobot_ethan/
python lerobot/scripts/control_robot.py     --robot.type=koch     --control.type=record     --control.fps=30     --control.single_task="example"     --control.repo_id="demo/example"     --control.num_episodes=1     --control.push_to_hub=true     --control.episode_time_s=50     --control.reset_time_s=10  

After recording the dataset, you can use the following command to visualize the dataset.

python lerobot/scripts/visualize_dataset_html.py   --repo-id demo/example

#### Important Note:
1. press **right key** for saving demo, **left** one for resetting the environment(if the demo fails to finish)
2. I suggest recording at least 50 episodes, with 10 episodes per location. Keep the cameras fixed and maintain consistent grasping behavior throughout the recordings.**REMEMBER** **,** **DATA QUALITY IS CRITICAL IN DATA COLLECTION.**
3. Place Camera to one in front, and one in top-down view
4. The object or target should be seen in **both camera** view as long as possible to keep the better data quality,if the robotic arm block the object in several frames, it will cause poor performance in inference.

#### Known Issue
1. It's difficult to use more than two cameras in setting due to the communication speed, the hub is too slow to send camera info if you plug more than one camera.
2. You have to insert THE camera in your PC first(phone camera), and then insert the other one with hub(front camera).

## Train
After Collecting dataset, you are ready to train your own imitation learning model!

conda activate /home/bruce/anaconda3/envs/lerobot/
python lerobot/scripts/train.py   --dataset.repo_id=Your_huggingface_username/task_name   --policy.type=act   --output_dir=outputs/train/your_task_name   --job_name=your_task_name  --save_freq=2000   --device=cuda   --wandb.enable=true 

**Important Note:**
1. The default training step is 100K, it takes around 8hours to train.
2. Feel free to test the model performance in training process, you can keep the training terminal open, and create another terminal to test the current model performance.

## Inference
After training, it's time to test your own model!

conda activate /home/bruce/anaconda3/envs/lerobot_ethan/
python lerobot/scripts/control_robot.py \
  --robot.type=koch \
  --control.type=record \
  --control.fps=30 \
  --control.single_task="eval_example" \
  --control.repo_id="ethanCSL/eval_example" \
  --control.num_episodes=1 \
  --control.warmup_time_s=2 \
  --control.episode_time_s=300 \
  --control.reset_time_s=10 \
  --control.push_to_hub=true \
  --control.policy.path=outputs/train/example/checkpoints/last/pretrained_model/

**Important Note:**
1. **Every time before you run the inference code, you have to remove the (eval_your_task_name), which is saved in this directory!!!:**

/home/bruce/.cache/huggingface/lerobot/your_huggingface_user_name/

eg:

~/.cache/huggingface/lerobot/lalalala0620$ ls
eval_act_koch_lego  koch_blue_paper_tape  koch_pick_place_lego  koch_test  koch_yellow_paper_tape

in this case, delete eval_act_koch_lego, before running the inference mode.

## Experiment Result
### Lego_50
Task: lift lego block
Dataset: 50 demos
Success rate:10%
#### Takeaway:

### Lego_100
Task: lift lego block
Dataset: 50 demos
Success rate:15~20%
#### Takeaway

### Lego_100_class
Task: Seperate the lego with yellow and white, and put it into the bin.
Dataset: 100 demos
Success rate:
(70~80% to catch only lego block one by one to its bin)
(30~40% if two lego blocks put together)

#### Takeaway

### Lego_100_class_test2
Task: Seperate the lego with yellow and white, and put it into the bin.
Dataset: 100 demos
Success rate:10~15%
#### Takeaway
4/1 train 4/2 test 3
檔名：final_test
測資：80筆
描述： 黑白方塊放入白黃區域
直直：20筆
橫橫：20筆
左斜斜：10筆
右斜斜：10筆
左右斜斜：20筆
先夾白再夾黑
成功率:特定位置定擺法60%以上,（位置：左黑右黃並排放中間）,其他位置幾乎無法成功夾取（10%）

###
## Troubleshooting
#### Record data
If you fail to record dataset, the following things are the things you have to inspect.

**1. Permittion Issue**
You will have to give access to both camera and robotic arm in the first time they've been plugged in your pc.

sudo chmod 666 /dev/video*
sudo chmod 666 /dev/ttyUSB*

**2.Huggingface user**
If you have problem using your own huggingface account while running command, please double check you have give access by the generated write token.
(huggingface-cli login --token your_token --add-to-git-credential )

**3.Camera or motor permittion**
If you have problem setting the camera or motor, check the permittion or you are using incorrect motor or camera index.

Please check the camera and motor index
Check robot config:

cd lerobot/common/robot_devices/robots/configs.py

locate this part:

@RobotConfig.register_subclass("koch")
@dataclass
class KochRobotConfig(ManipulatorRobotConfig):
    calibration_dir: str = ".cache/calibration/koch"
    # `max_relative_target` limits the magnitude of the relative positional target vector for safety purposes.
    # Set this to a positive scalar to have the same value for all motors, or a list that is the same length as
    # the number of motors in your follower arms.   
    max_relative_target: int | None = None

    leader_arms: dict[str, MotorsBusConfig] = field(
        default_factory=lambda: {
            "main": DynamixelMotorsBusConfig(
                port="/dev/ttyUSB1",
                motors={
                    # name: (index, model)
                    "shoulder_pan": [1, "xl330-m077"],
                    "shoulder_lift": [2, "xl330-m077"],
                    "elbow_flex": [3, "xl330-m077"],
                    "wrist_flex": [4, "xl330-m077"],
                    "wrist_roll": [5, "xl330-m077"],
                    "gripper": [6, "xl330-m077"],
                },
            ),
        }
    )

    follower_arms: dict[str, MotorsBusConfig] = field(
        default_factory=lambda: {
            "main": DynamixelMotorsBusConfig(
                port="/dev/ttyUSB0",
                motors={
                    # name: (index, model)
                    "shoulder_pan": [1, "xl430-w250"],
                    "shoulder_lift": [2, "xl430-w250"],
                    "elbow_flex": [3, "xl330-m288"],
                    "wrist_flex": [4, "xl330-m288"],
                    "wrist_roll": [5, "xl330-m288"],
                    "gripper": [6, "xl330-m288"],
                },
            ),
        }
    )

    cameras: dict[str, CameraConfig] = field(
        default_factory=lambda: {
            "front": OpenCVCameraConfig(
                camera_index=6, #6
                fps=30,
                width=640,
                height=480,
            ),
            "phone": OpenCVCameraConfig(
                camera_index=0,
                fps=30,
                width=640,
                height=480,
            ),
        }
    )

Make sure the camera index for "laptop and phone" is compatible to your current setup.

Check camera index

ls /dev/video*

cameras: dict[str, CameraConfig] = field(
    default_factory=lambda: {
        "laptop": OpenCVCameraConfig(
            camera_index=0,
            fps=30,
            width=640,
            height=480,
        ),
        "phone": OpenCVCameraConfig(
            camera_index=1,
            fps=30,
            width=640,
            height=480,
        ),
    }

Check robotic arm index:

ls /dev/ttyUSB*

And modify USB port
Use this command:

python lerobot/scripts/find_motors_bus_port.py

leader_arms: dict[str, MotorsBusConfig] = field(
    default_factory=lambda: {
        "main": DynamixelMotorsBusConfig(
            port="/dev/ttyUSB1",
            motors={
                # name: (index, model)
                "shoulder_pan": [1, "xl330-m077"],
                "shoulder_lift": [2, "xl330-m077"],
                "elbow_flex": [3, "xl330-m077"],
                "wrist_flex": [4, "xl330-m077"],
                "wrist_roll": [5, "xl330-m077"],
                "gripper": [6, "xl330-m077"],
            },
        ),
    }
)

follower_arms: dict[str, MotorsBusConfig] = field(
    default_factory=lambda: {
        "main": DynamixelMotorsBusConfig(
            port="/dev/ttyUSB0",
            motors={
                # name: (index, model)
                "shoulder_pan": [1, "xl430-w250"],
                "shoulder_lift": [2, "xl430-w250"],
                "elbow_flex": [3, "xl330-m288"],
                "wrist_flex": [4, "xl330-m288"],
                "wrist_roll": [5, "xl330-m288"],
                "gripper": [6, "xl330-m288"],
            },
        ),
    }
)

## Train LeRobot with Franka emika panda dataset
**1. Navigate to the desired directory**

cd ~/.cache/huggingface/lerobot/StanleyChueh

**2. Clone from huggingface nyu franka dataset**

git clone https://huggingface.co/ethanCSL/three_cube_stack

**4.Start Training**
On 5090

conda activate lerobot_nn
cd lerobot_nn
lerobot-train   --dataset.repo_id=ethanCSL/pick_n_place_50   --policy.type=act   --output_dir=outputs/train/act_pick_n_place_50   --job_name=act_pick_n_place_50   --policy.device=cuda   --policy.repo_id=ethanCSL/act_policy  --dataset.video_backend=pyav

note: tested branch: 1786916a

cd lerobot
git checkout 1786916a

**5.Transfer files between PCs**
Make sure these two devices are in the same domain

rsync -avzP ~/.cache/huggingface/lerobot/ethanCSL/XXX target_name@target_ip:~/.cache/huggingface/lerobot/ethanCSL/XXX

# Franka Lerobot

## Try it now first!
Please make sure every terminal is in ROS1 noetic environment, and check is CSL-FET@TT.

source /opt/ros/noetic/setup.bash

### Connect to Franka
There're two ways to connect to Franka
1. Command line

sudo ip addr add 172.16.0.1/24 dev enxc4411e75389a
sudo ip addr flush dev enxc4411e75389a
sudo ip addr add 172.16.0.1/24 dev enxc4411e75389a
sudo ip link set enxc4411e75389a up
sudo ufw disable

2. Run the connection code

cd ~/franka_record
python connect_franka.py

### Launch Franka_ROS

cd ~/franka_ws
source /opt/ros/noetic/setup.bash
source devel/setup.bash
roslaunch franka_example_controllers cartesian_impedance_example_controller.launch robot_ip:=172.16.0.2 load_gripper:=true launch_rviz:=false 

### Control Robot

cd ~/avp_teleoperate_h1/teleop/lerobot_record
source /opt/ros/noetic/setup.bash
python roboticArm_pose_remote_threshold_ros1_test.py 

### Vision Pro Control

cd ~/avp_teleoperate_h1/teleop/lerobot_record
source /opt/ros/noetic/setup.bash
python teleop_arm_pose_threshold_ros1_test.py 

If you have this error when running the above code:

> [Errno 98] error while attempting to bind on address ('0.0.0.0', 8012): address already in use
> Check PID
> ```
> sudo lsof -i :8012
> ```
> Kill PID
> ```
> sudo kill -9 PID
> ```

### LeRobot Record with Franka
#### Record

cd ~/franka_record/stanley_record
source /opt/ros/noetic/setup.bash
python record_ros1_test.py --single_task custom_task --repo_id your_huggingface_account/custom_task

> Please replace 'custom_task' and 'your_huggingface_account/custom_task' to the actual task and repo_id you want.

#### Visualization

Push to huggingface

cd ~/franka_record/
python push_to_hub.py 

Note:
> If you have trouble pushing to huggingface, please make sure you have the authentication set in your PC, please follow the instruction below:
> https://hackmd.io/-TIq0K1NROibtOAh4-sfSQ?both=&stext=1313%3A28%3A0%3A1751419547%3Aeqh_-6
>
Now you can do the visualization!

conda activate lerobot
python lerobot/scripts/visualize_dataset_html.py \
  --repo-id your_huggingface_user_name/repo_id


##### If you don't have your own dataset recorded, please try the following command to see the example:

python lerobot/scripts/visualize_dataset_html.py \
  --repo-id StanleyChueh/franka_lerobot_red_cube

-----------------------------
**Note:**
> Please make sure you have dataset in
> ```
> ls ~/.cache/huggingface/lerobot/StanleyChueh/franka_lerobot_red_cube
> ```
> Otherwise you should re-clone it:
> ```
> cd ~/.cache/huggingface/lerobot/StanleyChueh
> git clone https://huggingface.co/datasets/StanleyChueh/franka_lerobot_red_cube
> ```

### LeRobot Replay with Franka
1. Launch Franka ROS

cd ~/franka_ws
source /opt/ros/noetic/setup.bash
source devel/setup.bash
roslaunch franka_example_controllers cartesian_impedance_example_controller.launch robot_ip:=172.16.0.2 load_gripper:=true launch_rviz:=false 

2. Set initial position and switch to impedance control

cd ~/avp_teleoperate_h1/teleop/lerobot_record
source /opt/ros/noetic/setup.bash
python roboticArm_pose_remote_threshold_ros1_test.py 

> ## Note!!!!!!!!!:
> [INFO] [1751621729.042889]: Switched to cartesian impedance controller.
> Once this message pops out, you can shut this code down.
3. Replay Episode

 python ~/franka_record/stanley_record/tools/replay_ros1_v2_quat.py 

### LeRobot Training with Franka

#### ACT

conda activate lerobot
cd ~/CSL/lerobot_new
python -m lerobot.scripts.train --policy.type=act --dataset.repo_id=user_name/repo_name --output_dir=outputs/train/your_task_name

#### SmolVLA

python train.py   --policy.path=lerobot/smolvla_base   --dataset.repo_id=ethanCSL/smolvla_multiblock   --batch_size=16   --steps=20000   --output_dir=outputs/train/svla_multiblock   --job_name=my_smolvla_training   --policy.device=cuda   --wandb.enable=false --policy.repo_id=svla_multiblock

### LeRobot Evaluation with Franka

#### On Agx Orin
Launch socket server

conda activate lerobot
python lerobot/lerobot/scripts/eval_franka_socket_v2.py

#### On control PC(10.100.4.119), On bruce PC(10.100.4.42)

##### Launch Franka_ROS

cd ~/franka_ws
source /opt/ros/noetic/setup.bash
source devel/setup.bash
roslaunch franka_example_controllers cartesian_impedance_example_controller.launch robot_ip:=172.16.0.2 load_gripper:=true launch_rviz:=false 

##### Control robot

cd ~/avp_teleoperate_h1/teleop/lerobot_record
source /opt/ros/noetic/setup.bash
python roboticArm_pose_remote_threshold_ros1_test.py 

> ## Note!!!!!!!!!:
> [INFO] [1751621729.042889]: Switched to cartesian impedance controller.
> Once this message pops out, you can shut this code down.

##### Publish camera topic

python ~/franka_record/stanley_record/image_publisher_SA.py 

##### Launch socket server & client

 python evaluation.py --server-ip 10.100.4.42 --target-hz 10

 python franka_socket_test_stability_test.py --ckpt-path /home/bruce/CSL/lerobot_nn/src/lerobot/scripts/outputs/train/act_pick_n_place_50/pretrained_model/ --eval-freq 10

### 📁 Project Structure

avp_teleoperate_h1/
├── act/
├── assets/
├── img/
├── scripts/
├── teleop/
│   ├── teleop_arm_pose_threshold_ros1_test.py        # 🖐️ Hand tracking via Vision Pro
│   └── roboticArm_pose_remote_threshold_ros1_test.py # 🤖 Franka Panda control

franka_ros/
└── src/
    └── franka_ros/
        └── franka_example_controllers/
            └── launch/
                └── cartesian_impedance_example_controller.launch  # Launch file for Cartesian impedance control

franka_record/
├── franka_dataset.py   # LeRobot-compatible dataset structure
└── record_ros1.py      # ROS1-based recording script