# Camera LiDAR Calibration

## DeepI2P

- Frustum Classification - 1 if point is in FOV
- Classification of which 32x32 grid the point belongs to
- Based on the FOV classification - Use Gauss Newton to optimize the pose
- Cost Function $$\left\{
\begin{array}{lr} r_i^0 , & \text{if } \hat{l_i^c} = 0\\ r_i^1 , & \text{if } \hat{l_i^c} = 1 \end{array} \right\}$$ $$r_i^0 = \left( u(p^{\prime}_{x_i};W) + u(p^{\prime}_{y_i};H) \cdot \mathbb{1}(p^{\prime}_{x_i},p^{\prime}_{y_i}, z^{\prime}_i;H,W) \right)$$ $$r_i^1 = g(p^{\prime}_{x_i};W) + g(p^{\prime}_{y_i};H) + h(z^{\prime}_i)$$ $$u(p^{\prime}_{x_i};W) = \frac{W}{2} - \Big|p^{\prime}_{x_i} - \frac{W}{2}\Big| $$ $$g(p^{\prime}_{x_i};W) = max(-p^{\prime}_{x_i}, 0) + max(p^{\prime}_{x_i} - W, 0)$$ $$h(z^{\prime}_i) = \alpha \cdot max(-z^{\prime}_i, 0)$$

### Experiments
- Direct Regression - Performs bad, use angle-axis and translation vector regression. Can be improved via a better loss function.$$||Log_{\mathit{SE}(3)}(T_{gt}^{-1}T)||$$
- Monodepth + ICP - Treat it as a Pointcloud Registration problem
- 2D-3D Matchnet - Do no Rotation augmentations.

### Shortcomings
- Poor Results
- Rudimentary Cost Function

## CorrI2P - [Link](https://arxiv.org/pdf/2207.05483.pdf)
Uses Features from the network as descriptors for Pose Estimation. Once overlapping region is decided, use nearest neighbour to find closes features.


The confidence score is used to filter out the overlapping region. Features for the overlapping regions are trained to be similar across camera and LiDAR.

- Use Negative Margin loss for the embeddings

## Shortcomings
- Errors are still around 0.75m which is quite high.

# Ideas
- Predict Normals of LiDAR in the camera frame.
- DROID-SLAM like iterative optimization for LiDAR location on the pixel. - [Link](https://arxiv.org/abs/2108.10869)