コンピュータビジョン：認識から幾何学まで
===

Assignment1 : Image Corner Detection and Filtering
===

## Part 1: Image Corner Detection
* **Implement Harris corner detector**
> A combined corner and edge detector, Chris Harris & Mike Stephens, Plessey Research Roke Manor, United Kingdom, The Plessey Company plc. 1988

* **Moravec Corner Detector**
For a corner, shifting a window in any direction should give a large change in intensity.

* **Moravec Corner Detector**
For a given patch (x, y) and displacement (u, v), the difference function can be written as

$E(u,v)=\sum_{(x,y)}w(x,y)[I(x+u,y+v)-I(x,y)]^2$
- $w(x,y)$：window function
- $I(x+u,y+v)$: shifted intensity
- $I(x,y)$: intensity
Corner response for the center pixel is defined as $R = \min_{(u,v)} E(u,v)$.
$(u,v)={(1,0),(1,1),(0,1),(-1,1)}$ for Moravec corner detector
* **Harris Corner Detector**
Moravec model only considers a set of shifts at every 45 degree, while Harris model considers all small shifts by using Taylor’s expansion.

$E(u,v)=\sum_{(x,y)}w(x,y)[I(x+u,y+v)-I(x,y)]^2$
Small motion assumption + Taylor Series Expansion [1]
$\Longrightarrow E(u,v)\approx\ \begin{pmatrix} u & v \end{pmatrix} M \begin{pmatrix} u \\ v \end{pmatrix}$
$M = \sum_{x,y} w(x,y) \begin{pmatrix} I_x I_x & I_x I_y \\ I_y I_x & I_y I_y \end{pmatrix}$
$I_x$ derivative of intensity along x
$I_y$ derivative of intensity along y
Definition of $I_x$ and $I_y$ in this assignment:
|62|75|38|
|--|--|--|
|26|54|97|
|57|27|5|
For the center pixel:
$I_x = 1 \cdot 26 + (-1) \cdot 97 = -71$
$I_y = 1 \cdot 75 + (-1) \cdot 27 = 48$
* **Harris Corner Detector**
Important property of a 2-by-2 matrix $M$

* let $\lambda_1$ and $\lambda_2$ as eigen values of M.
* $\lambda_1$ and $\lambda_2$ are small, thre region is flat.
* $\lambda_1 >> \lambda_2$ or vice versa, the region is edge.
* $\lambda_1$ and $\lambda_2$ are large and $\lambda_1 ~ \lambda_2$, the region is a corner.
* **Harris Corner Detector**
* Harris corner response equation
*$R = \lambda_1\lambda_2-k(\lambda_1+\lambda+2)^2=det(M)-k\cdot trace(M)^2$
*$R = \lambda_1\lambda_2 / (\lambda_1+\lambda_2)=det(M)/ trace(M)$ We use this response equation in this assignment.
* **Harris Corner Detector**
* **Step1** : Smooth the image by Gaussian Kernel.
* Function : $cv2.GaussianBlur (kernel=3, sigma=1.5)$

    cv.GaussianBlur (src, dst, ksize, sigmaX, sigmaY, borderType = cv.BORDER_DEFAULT)

**Parameters**
**src** input image; the image can have any number of channels, which are processed independently, but the depth should be CV_8U, CV_16U, CV_16S, CV_32F or CV_64F.
**dst** output image of the same size and type as src.
ksize blurring kernel size.
**sigmaX** Gaussian kernel standard deviation in X direction.
**sigmaY** Gaussian kernel standard deviation in Y direction; if sigmaY is zero, it is set to be equal to sigmaX, if both sigmas are zeros, they are computed from ksize.width and ksize.height, to fully control the result regardless of possible future modifications of all this semantics, it is recommended to specify all of ksize, sigmaX, and sigmaY.
**borderType** pixel extrapolation method(see cv.BorderTypes).

* $M = \sum_{x,y} w(x,y) \begin{pmatrix} I_x I_x & I_x I_y \\ I_y I_x & I_y I_y \end{pmatrix}= \sum_{x,y} w(x,y) \begin{pmatrix} I_{xx} & I_{xy} \\ I_{yx} & I_{yy} \end{pmatrix}= \begin{pmatrix} \sum_{x,y} w(x,y)I_{xx} & \sum_{x,y} w(x,y)I_{xy} \\ \sum_{x,y} w(x,y)I_{yx} & \sum_{x,y} w(x,y)I_{yy} \end{pmatrix}= \begin{pmatrix} S_{xx} & S_{xy} \\ S_{yx} & S_{yy} \end{pmatrix}$
* **Step2** : Calculate $I_x$, $I_y$ (First derivateive of image along x and y axis.)
* Function : $cv2.filter2D (kernel)$

    # kernel=[[1.,0.,-1.]] for Ix or [[1.],[0.],[-1.]] for Iy
    dst = cv.filter2D ( src, ddepth, kernel[, dst[, anchor[, delta[, borderType]]]]	)

* **Step3** : Calculate $I_{xx} = I_x \cdot I_x$, $I_{xy} = I_x \cdot I_y$, $I_{yy} = I_y \cdot I_y$
* **Step4** : Compute $S_{xx}$, $S_{xy}$, $S_{yy}$ (weighted summation of $I_{xx},I_{xy},I_{yy}$ in neighbor pixels)
* Function : $cv2.GaussianBlur (kernel=3, sigma=1.)$
* **Step5** : Compute the determinance and trace of matrix $M$
* **Step6** : Compute the response of the detector by determinance / (trace + 1e-12)
* [https://docs.opencv.org/3.4/d4/d86/group__imgproc__filter.html](https://)
* **Harris Corner Detector**
* Post Processing
* Thresholding
* Find local maximum
* **Harris Corner Detector**
* Threshold
* The pixels whose responses > threshold would be selected as candidates
* Find local maximum
* The candidates whose responses > all its 5x5 neighbors (24 pixels totally) are recognized as final corner.
* Need zero padding (width of 2) to maintain the output size.
* **Assignment Description**
* part1/main.py
* Read image, execute Harris corner detector, visualize results, ... etc

import numpy as np
import cv2
import argparse
from HCD import Harris_corner_detector


def main():
    parser = argparse.ArgumentParser(description='main function of Harris corner detector')
    parser.add_argument('--threshold', default=100., type=float, help='threshold value to determine corner')
    parser.add_argument('--image_path', default='./testdata/1.png', help='path to input image')
    args = parser.parse_args()

    print('Processing %s ...'%args.image_path)
    img = cv2.imread(args.image_path)
    img_gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY).astype(np.float64)

    ### TODO ###


if __name__ == '__main__':
    main()

* part1/HCD.py
* Implement Harris corner detector, including two functions
* detect_harris_corners: compute the corner response for input grayscale image
* post_processing : detect the corner for the giving response map.
* The output format should be "list of list"

import numpy as np
import cv2
import matplotlib.pyplot as plt


class Harris_corner_detector(object):
    def __init__(self, threshold):
        self.threshold = threshold

    def detect_harris_corners(self, img):
        ### TODO ####
        # Step 1: Smooth the image by Gaussian kernel
        # - Function: cv2.GaussianBlur (kernel = 3, sigma = 1.5)

        # Step 2: Calculate Ix, Iy (1st derivative of image along x and y axis)
        # - Function: cv2.filter2D (kernel = [[1.,0.,-1.]] for Ix or [[1.],[0.],[-1.]] for Iy)

        # Step 3: Compute Ixx, Ixy, Iyy (Ixx = Ix*Ix, ...)

        # Step 4: Compute Sxx, Sxy, Syy (weighted summation of Ixx, Ixy, Iyy in neighbor pixels)
        # - Function: cv2.GaussianBlur (kernel = 3, sigma = 1.)

        # Step 5: Compute the det and trace of matrix M (M = [[Sxx, Sxy], [Sxy, Syy]])

        # Step 6: Compute the response of the detector by det/(trace+1e-12)

        return response
    
    def post_processing(self, response):
        ### TODO ###
        # Step 1: Thresholding

        # Step 2: Find local maximum

        return local_max

* **Assignment Description**
* part1/eval.py (DO NOT EDIT this file)
* Evaluate the correctness of the output of Harris corner detector
* TA will run this file to score upload code.
* When testing your code, we will assign different arguments, like threshold, and corresponding ground truth file.

import numpy as np
import cv2
import argparse
import pickle
from HCD import Harris_corner_detector


def main():
    parser = argparse.ArgumentParser(description='evaluation function of Harris corner detector')
    parser.add_argument('--threshold', default=100., type=float, help='threshold value to determine corner')
    parser.add_argument('--image_path', default='./testdata/ex.png', help='path to input image')
    parser.add_argument('--gt_path', default='./testdata/ex_gt.pkl', help='path to ground truth pickle file')
    args = parser.parse_args()

    img = cv2.imread(args.image_path)
    img_gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY).astype(np.float64)

    # create HCD class
    HCD = Harris_corner_detector(args.threshold)
    
    response = HCD.detect_harris_corners(img_gray)
    result = HCD.post_processing(response)

    with open(args.gt_path, 'rb') as f:
        gt = pickle.load(f)

    gt_match, out_match = 0, 0
    for gt_coord in gt:
        count = result.count(gt_coord)
        if count >= 1:
            gt_match += 1
            out_match += count
    print('[Error] Result unmatch: %d\n[Error] Ground truth unmatch: %d'%
           (len(gt)-gt_match, len(result)-out_match))


if __name__ == '__main__':
    main()

* **Assignment Description**
* PART1/testdata/
* Include 1 example image with ground truth + other 3 images without groundtruth
* **Assignment Description**
* Recommended steps
* Implment Harris corner detector in HCD.py
* Use eval.py to evaluate your HCD.py
* By
` python3 eval.py --image_path 'testdata/ex.png' --gt_path 'testdata/ex_gt.pkl'`
* The result and Groud Truth unmatch should be both 0, as
* Finish remaining code in main.py if needed.
* **Testdata**
* 1.png

* 2.png

* 3.png

* ex_gt.png

* ex.png

* **Reference**
* Harris, Christopher G., and Mike Stephens. "A combined corner and edge detector", 1988.
* NTU VFX course slide https://www.csie.ntu.edu.tw/~cyy/courses/vfx/19spring/lectures/handouts/lec06_feature.pdf
* OpenCV-Python Tutorial: Harris Corner Detection https://docs.opencv.org/master/dc/d0d/tutorial_py_features_harris.html
* Wikipedia: Corner detection https://en.wikipedia.org/wiki/Corner_detection
---
## Part 2: Image Filtering
* Implement bilateral filter
* Advanced color-to-gray conversion