---
# System prepended metadata

title: Camera LiDAR Calibration

---

# Camera LiDAR Calibration

## DeepI2P
![](https://hackmd.io/_uploads/Sy-mWykBh.png)

- Frustum Classification - 1 if point is in FOV
- Classification of which 32x32 grid the point belongs to
- Based on the FOV classification - Use Gauss Newton to optimize the pose
    - Cost Function $$\left\{
    \begin{array}{lr} r_i^0 , & \text{if } \hat{l_i^c} =  0\\ r_i^1 , & \text{if } \hat{l_i^c} =  1 \end{array} \right\}$$ $$r_i^0 = \left( u(p^{\prime}_{x_i};W) + u(p^{\prime}_{y_i};H) \cdot \mathbb{1}(p^{\prime}_{x_i},p^{\prime}_{y_i}, z^{\prime}_i;H,W) \right)$$ $$r_i^1 =  g(p^{\prime}_{x_i};W) + g(p^{\prime}_{y_i};H) + h(z^{\prime}_i)$$ $$u(p^{\prime}_{x_i};W) = \frac{W}{2} - \Big|p^{\prime}_{x_i} - \frac{W}{2}\Big| $$ $$g(p^{\prime}_{x_i};W) = max(-p^{\prime}_{x_i}, 0) + max(p^{\prime}_{x_i} - W, 0)$$ $$h(z^{\prime}_i) = \alpha \cdot max(-z^{\prime}_i, 0)$$
    
### Experiments
- Direct Regression - Performs bad, use angle-axis and translation vector regression. Can be improved via a better loss function.$$||Log_{\mathit{SE}(3)}(T_{gt}^{-1}T)||$$
- Monodepth + ICP - Treat it as a Pointcloud Registration problem
- 2D-3D Matchnet - Do no Rotation augmentations.

### Shortcomings
- Poor Results
- Rudimentary Cost Function


## CorrI2P - [Link](https://arxiv.org/pdf/2207.05483.pdf)
Uses Features from the network as descriptors for Pose Estimation. Once overlapping region is decided, use nearest neighbour to find closes features. 

![](https://hackmd.io/_uploads/SyFF7tcBn.png)
![](https://hackmd.io/_uploads/SyZjmK5Hh.png)

The confidence score is used to filter out the overlapping region. Features for the overlapping regions are trained to be similar across camera and LiDAR.

- Use Negative Margin loss for the embeddings

## Shortcomings
- Errors are still around 0.75m which is quite high.





# Ideas
- Predict Normals of LiDAR in the camera frame.
- DROID-SLAM like iterative optimization for LiDAR location on the pixel. - [Link](https://arxiv.org/abs/2108.10869)
