# Image Rectification

<style>
figure {
border: 1px #cccccc solid;
padding: 4px;
margin: auto;
text-align: center;
}

figcaption {
background-color: black;
color: white;
font-style: italic;
padding: 1px;
text-align: center;
}
</style>

Image recification is to make epipolar lines epipolar lines are perfectly aligned horizontally between stereo image.

<figure style="text-align: center">
<img src="https://hackmd.io/_uploads/r1Gz-sxO6.png"/>
</figure>

Recall that in [epipolar geometry post](https://hackmd.io/@jackyyeh/Hkk6MrqIT), we mentions two goals for epipolar geometry: Solve correspondence problem & Triangulation. In fact, we can solve these two problems more easily after image planes are rectified(aligned).

## Easier to reconstruct depth
<figure style="text-align: center">
<img src="https://hackmd.io/_uploads/HJnz0qeuT.png" width=700x />
<figcaption>Top-view stereo vision</figcaption>
</figure>

Triangle $OPO'$ Triangle $POP'$ arer similar. Therefore,
$$
depth = \frac{Baseline \cdot f}{p_{u} - p_{u'}} = \frac{Baseline \cdot f}{Disparity}
$$

Note that disparity the apparent motion of objects between a pair of stereo images.

## Easier to solve correspondense problem
<figure style="text-align: center">
<img src="https://hackmd.io/_uploads/rJd6JseOT.png" width=700x />
</figure>

After rectifying two stereo images, in theory it is guaranteed the ==corresponding feature points will appear in the same horizontal lines($p_{v} = p'_{v}$)==.

Here's why.

Let us first compute the Essential matrix $E$ in the case of parallel image planes. We can assume that the two cameras have the same $K$ and that there is no relative rotation between the cameras ($R = I$). In this case, let us assume that there is only a translation along the x-axis, giving $t = (t_{x}, 0, 0)$. This gives

$$
E = [t_{\times}]R =
\begin{bmatrix}
0 & 0 & 0 \\
0 & 0 & -t_{x} \\
0 & t_{x} & 0
\end{bmatrix}
$$

After $E$ is known, we can compute the epipolar line associated with point $p'$:
$$
l = Ep'=
\begin{bmatrix}
0 & 0 & 0 \\
0 & 0 & -t_{x} \\
0 & t_{x} & 0
\end{bmatrix}
\begin{bmatrix}
u' \\
v' \\
1
\end{bmatrix} =
\begin{bmatrix}
0 \\
-t_{x} \\
t_{x}v'
\end{bmatrix}
$$

Then according to epipolar constraint $p^{T}Ep' = 0$, we derive $v = v'$, representing that $p$ and $p'$ share the same $v$-coordinate.

## So how to rectify?

Find two homography matrics $H, H'$ to make epipolar lines horizontal for two images.

<figure>
<img src="https://hackmd.io/_uploads/rktmIvxd6.png" width="400"/>
</figure>

==TODO (warning: a lot of math)==
([ref](https://web.stanford.edu/class/cs231a/course_notes/03-epipolar-geometry.pdf))

## Depth map

With rectified image pair, correspondence problem can be reduced to matching problem within a horizontal line. After constructing corresponding pairs, we can easily recover disparity map on the right using the following formula:

$$
depth = \frac{Baseline \cdot f}{p_{u} - p_{u'}}
$$

## Baseline distance selection
<figure>
<img src="https://hackmd.io/_uploads/r1L6PcjL6.png"/>
</figure>

- Too large: close objects can not be searched for & prone to occlusion.
- Too small: a large depth error.

## Supplement
- 3D movies
3D movies present different images to our left and right eyes, and this differential input allows our brain to process the images into a three-dimensional representation.

## References
- https://github.com/polygon-software/python-visual-odometry/blob/master/Chapter%208%20-%20Stereovision%20Trinagulation%20Feature%20Correspondance%20Disparity%20Map.ipynb
- https://web.stanford.edu/class/cs231a/course_notes/03-epipolar-geometry.pdf
- https://www.youtube.com/watch?v=My0I6fPfpEI&list=PLFI1Cd4723_SQ-h8W0kT3igOoUCYjWRHG&t=3131s