# Optical Flow
### Reference: FPCV Youtube

Assumption:
- Brightness constancy
- Small $\delta x, \delta y, \delta t$; so that we can use Taylor's approx

## Brightness constancy:
$$
I(x+\delta x, y+\delta y, t+\delta t) = I(x,y,t) \\
\implies
\frac {\partial I}{\partial x} \delta x + \frac {\partial I}{\partial y} \delta y + \frac {\partial I}{\partial t} \delta t = 0 \\
\implies
I_xu + I_yv + I_t = 0
$$

At any point we know $I_x, I_y, I_t$ and want to find the optical flow $(u,v)$.
Two variables, One equation!

We know $(u,v)$ is on a line but we dont know where on the line

We can only determine the normal flow:
$$
u_n = \frac {\vert I_t \vert (I_x,I_y)}{\sqrt{I_x^2+I_y^2}}
$$

This is called aperture problem because this is the flow visible through an aperture. The tangent component is not visible through the aperture.

## Lucas Kanade
### LK assumption
Optical flow is constant within a small window.

So, we now have a system of equations of size n*n where n is the size of window

$$
\begin{bmatrix} I_x(1,1)&I_y(1,1) \\ I_x(1,2)&I_y(1,2) \\ I_x(1,3)&I_y(1,3) \\..\\ I_x(n,n)&I_y(n,n)\end{bmatrix}
\begin{bmatrix} u \\ v \end{bmatrix} =
\begin{bmatrix} I_t(1,1) \\ I_t(1,2) \\ I_t(1,3) \\ .. \\ I_t(n,n) \end{bmatrix}
$$

This is equivalent to $Au=b$.
We can pose this as
$$
\min_u (Au-b)^T(Au-b)
$$
Solution to this is $$u=(A^TA)^{-1}A^Tb$$

$$
(A^TA) =
\begin{bmatrix}
\sum_w I_x^2 & \sum_w I_xI_y \\
\sum_w I_xI_y & \sum_w I_y^2
\end{bmatrix}
$$

$$
(A^Tb) =
\begin{bmatrix}
\sum_w I_xI_t \\
\sum_w I_yI_t
\end{bmatrix}
$$

Now we need that $A^TA$ matrix be well-conditioned ie $det(A) > 0$

$\lambda_1,\lambda_2 = eigval(A^TA)$. Then we want that
$\lambda_1>\epsilon$,
$\lambda_2>\epsilon$ and
$\lambda_1>>>\lambda_2$ should not be there.

## Large motion
Notice that optical flow reduces by half if you decrease image resolution. So even for large motion, at very low resolution, the Taylor assumption will hold.

### Coarse to fine estimation
Estimate OF at lower resolution. Upscale OF. Warp image as per OF. Get residual OF. Continue till you reach original resolution.

### Alternative approach
Template matching to directly get $(u,v)$ for keypoint.