---
# System prepended metadata

title: 'NeRF in the Palm of Your Hand: Corrective Augmentation for Robotics via Novel-View Synthesis'

---

# NeRF in the Palm of Your Hand: Corrective Augmentation for Robotics via Novel-View Synthesis

## Introduction

This paper is about improving robotic arm grasping using NeRF to augment the training dataset. NeRF is a Neural Network method used to synthesis complex 3d scene. The input of this NN is a position and a viewing direction and the output is a pair of volume density and emitted radiance (color).

The goal of this paper is to build a data augmentation pipeline for synthesis new eye-in-hand demonstration using perturbed poses and corrective action.

This pipeline uses only rgb images, is offline and but still has a closed loop and uses 6DOF of the robotic arm (while other methods are limited or need online policy).

## How SPARTN works

SPARTN (Synthetic Perturbations for Augmenting Robot Trajectories via NeRF) is the algorithm developed in this paper.
The algorithm works by creating a NeRF model for each trajectory present in the dataset.

A trajectory is a set of pairs of state and action that describe a full motion from the initial pose of the robot to the grasping pose. A state is composed by an image and a pose, while the action is a what the robot arm has to do in that particular state to go to the next state.

Then for each pair of state, action in the trajectory SPARTN create X new pairs to augment the dataset.
- To create a new state the idea is to perturb the pose then use it to generate a new high-fidelity image using as the input of the particular NeRF.
- The correct action creation is simple because is done by computing the inverse kinematic that allow the robot arm to go from the perturbed pose to the next non perturbed pose in the true trajectory.


## Policy training

In this paper, Behavior Cloning is use to learn a grasping policy. This method is offline and use supervised learning to train a policy by mimiching an expert.


## Implementation details

### Static scene

The NeRF model assumes that the scene is static so to perform the learning from a dataset of a grasping task we can use only a subset of a trajectory where there is not robot-object interaction. A good assumption in grasping tasks is to use all the timesteps before the closing of the gripper.

### Masking Out the Robot Gripper

Another trick used to not break the static scene constraint is to mask out the gripper from the images. Since the gripper remains in a fixed position in the images, we create a mask. So utilizing the mask, we can exclude rays associated with gripper pixels during the training.

## Result

We ran some test in both a simulated environments and in a real ones.
We compare serveral different methods and obtain the best result in every scenario compared to similar offline algorithm. Only online policy methods obtain better result but they require interaction with the environment and so more expert demonstrations and time.

In particular on the real world grasping tests we obtain a 60% increase in performance compared to the standard Behavior Cloning.

## Conclusion

SPARTN is a very good solution compared to other method of data agumentation like cropping or color changing.

The real world test show how SPARTN is more reactive to changes in the environment and it also capable of recovering from a failure picking.

However it has some limitations. Standard NeRF requires a static scene and the process of training every single demonstrations is very expensive (we took 7 hours to train 2500 NeRF using 4 GPUs). Another limitation is that the noise used to create a new pose is tuned for the specific robot arm and so is not general.
