# CS445 Proj5 Report
###### tags: `FA19`

Wanxian Yang (wyang52)

#### Bells & Whistles (EC)
- [x] Process one more videos

### Part 1: Stitch Two Key Frames
For this part, I simply solve for `H` via SVD.

### Part 2: Panorama Using Five Key Frames
This part is done in a similar manner with Part 1, except for that since frame 90 and 810 is very different from frame 450, they are first mapped to frame 270 and 630 correspondingly, and then mapped to fram 450.

### Part 3: Map The Video To The Reference Plane
{%youtube MrTajgNIyI4 %}
Mapping all frames to frame 450 is done in two steps: first map frame 1-450 to 450, and then map frame 451-900 to 450.

For the mapping of frame 451-900, I did not reuse the `originTranslation` from the last mapping (i.e. the mapping of frame 1-450). This results in an incorrect shift of the frames. To correct this, I simply apply the translation of frame 1 to all the output frames of the frame 451-900.

### Part 4: Create Background Panorama
This part is done with two steps: 1. create masks for each of the 900 frames; 2. compute the background panorama using median, where the median is computed seperately for each chanel.

I computed the background panorama using the median of the overlapping pixels from the last part (where we produced 900 projected frames). The only complication is that in each of the 900 projected frames, there are both image pixels and black background pixels. When calculating the median, we do not want to take into account the background pixels (or else the output panorama is likely just a black canvas). This issue is solved by using a mask for each of the projected frames. For each pixel position of the output panorama, we only take into account the pixels of those projected frames where the corresponding mask has a `True` value at that position.

For a projected image, the corresponding mask is produced as follow. First, we create an all-white image with the same size of the original frame. Then, we apply the corresponding homography of that frame (the homographies are produced in Part 3) to the white image. The resulting image is effectively a mask for the corresponding projected frame.

### Part 5: Create Background Movie
{%youtube _8vW_p7BpZk %}
To produce a background movie, we first produce all 900 frames of the movie, and then produce a video using these 900 frames.

To produce the 900 frames, we need the background panorama produced from Part 4, and the homographies calculated in Part 3. To obtain a frame, we simply need to apply the __inverse__ of the corresponding homography to the panorama.

### Part 6: Create Foreground Movie
{%youtube 3T3fuo97v1M %}
To create a foreground movie, we first produce all 900 frames of the movie, and then produce a video using these 900 frames.

To produce the 900 frames, we need the 900 background frames produced from Part 5, and the 900 original frames. For each frame, to find the foreground pixels, we simply mask in the pixels in the original frame that are significantly different from the corresponding background frame.

How the difference is defined: I simply use the largest difference of each of the three chanels as the difference of the pixels. For example, let $p_1, p_2$ be two pixels, where $p_1 = (1,1,1), p_2 = (10, 20, 30)$, then the difference of $p_1$ and $p_2$ is $30-1 = 29$.

The threshold of 'significance' is a free parameter to be tuned. In my case, I found $50$ to be a better value for the parameter.

Note that it is important to have scaled the frames to the same scale before calculating the difference.

# Bells & Whistles
## Process One More Video
### Original Video
{%youtube 4jgPoDUL3WA %}
This video is converted into individual frames using a function in `utils.ipynb`

### Part 1: Stitch Two Key Frames
This part is done similarly as in the core assignment

### Part 2: Panorama Using Five Key Frames
This part is done similarly as in the core assignment, except that I get to choose which five frames I like to use.

### Part 3: Map The Video To The Reference Plane
{%youtube YGj3m1tUhqs %}
This part is done similarly as in the core assignment, except that since I only have 600-ish frames to process, instead of processing them in two steps, I only needed to do it one-pass.

### Part 4: Create Background Panorama
This part is done similarly as in the core assignment.

### Part 5: Create Background Movie
{%youtube 3Ie4tEbb2Hs %}
This part is done similarly as in the core assignment.

### Part 6: Create Foreground Movie
{%youtube 8-XeS3z9qJo %}
This part is done similarly as in the core assignment, except that I used a different threshold, which is 35.