# Exercise VI - Autonomous Lane Navigation Part 1 of 4

Diese Übung zielt darauf ab eine Einführung in das Arbeiten mit der Computer Vision Software OpenCV zu geben. Weiterhin ist Ziel dieser Übung sich mit dem HSV Color Space vertraut zu machen und zu verstehen welche Vorteile die Umwandlung des RGB-Farbraums in den HSV-Farbraum hat. Eine solche Umwandlung der Farbräume wird praktische Anwendung finden und an einem Beispiel erprobt werden. Auch der Begriff des Masking ist Bestandteil dieser Übung und soll den Kursteilnehmern einen tieferen Einblick in die Welt der Computer Vision und hier speziell in das angewandte Gebiet der Lane Navigation geben.

## Introduction to OpenCV, the HSV Color-Space and Masking

Many 2018 and newer vehicles are equipped with a feature called Lane Keep Assist System (LKAS). This LKA is still a relatively new feature that uses a camera on the windshield to identify a lane line. Using this feature, the car steers and is kept in the center of the lane. In this course, we also want to create a vehicle that can handle the features of autonomous driving. In this course, we also want to create a vehicle that can handle the features of autonomous driving. For this purpose, we use computer vision.

"Computer Vision is a field of computer science that deals with the extraction and processing of structured information from mostly unstructured image data. OpenCV (Open Source Computer Vision) is an open source computer vision and machine learning software library with more than 2500 optimized algorithms and is used in this course to process the input images, detect lane lines, and create a representation of the lane."

Quelle: @Online {AboutOpenCV-2022-01-24, title = {About - OpenCV}, date = {2022-01-24}, year = {2022}, file = {:./references/about-.html:html}, url = {https://opencv.org/about/}, urldate = {2022-01-24}}

## Key Objectives

The goal of this exercise is to learn about the essential components of the Lane Keep Assist system. Furthermore, the OpenCV computer vision package will be explored and an insight into it will be gained.

Another goal of this exercise is to learn about and understand the Hue Saturation Value (HSV) color space. Part of this exercise will also be to understand and apply the conversion from RGB color value to HSV color value. In addition, the advantages of the different color representations can be explained and applied to the course objectives.

## Accompanying Theory

**Lane Keep Assist System**
- A lane keep assist system has two components:
- Perception (lane detection) and
- Path/Motion Planning (steering)

The role of the lane keep assistant is to transform a video of the road into the concrete coordinates of the recognized lane lines. One way to implement this is to use the computer vision package OpenCV. We will first identify the lane lines in a single picture of our road. As soon as we are able to do this, the identification of lane lines in a video sequence is a simple repetition of the same steps for all video frames. To detect lane lines in a single frame, we need to become familiar with the use of the Hue Saturation Value color space.

#### Color spaces

Color spaces are a way to represent the color channels that exist in an image and give it that particular hue. There are different color spaces and each has its own meaning. Some of the most popular color spaces are RGB (red, green, blue), CMYK (cyan, magenta, yellow, black) and HSV (hue, saturation, value.

Source: @Online {FarbrumeinOpenCVPythonAcervoLima-2022-01-24, title = {Farbr\"aume in OpenCV | Python – Acervo Lima}, date = {2022-01-24}, year = {2022}, file = {:./references/farbraume-in-opencv-python-.html:html}, url = {https://de.acervolima.com/farbraume-in-opencv-python/}, urldate = {2022-01-24}}

#### Hue Saturation Value (HSV) Color Space

The Hue/Saturation/Value (HSV) Color Space is an alternative representation of the RGB color model, designed in the 1970s by computer graphics researchers to more closely align with the way human vision perceives color-making attributes. The HSV-representation models how colors appear under different light conditions.
Source:
@Online {FileHSVcolorsolidcylindersaturationgraypngWikimediaCommons-2022-01-11, title = {File:HSV color solid cylinder saturation gray.png - Wikimedia Commons}, date = {2022-01-11}, year = {2022}, file = {:./references/wiki-File:HSV_color_solid_cylinder_saturation_gray.png.html:html}, url = {https://commons.wikimedia.org/wiki/File:HSV_color_solid_cylinder_saturation_gray.png}, urldate = {2022-01-25}}

*HSV is a cylindrical coordinate representation of points in an RGB color model. The Hue (Angle) is the dominant color as perceived by an observer, the Saturation (Radius) is the amount of white light mixed with a hue and the Value (Height) is the chromatic notion of intensity. The additive primary and secondary colors (red, yellow, green, cyan, blue and magenta) and linear mixtures between adjacent pairs of them are arranged around the outside edge of the cylinder with saturation 1. Mixing these pure colors with black—producing so-called shades leaves the saturation unchanged. Only tinting reduces saturation.*

*Source: @Online {ColormodelsandcolorspacesProgrammingDesignSystems-2020-07-18, title = {Color models and color spaces - Programming Design Systems}, date = {2020-07-18}, year = {2020}, author = {Rune Madsen}, file = {:./references/color-color-models-and-color-spaces-index.html.html:html}, url = {https://programmingdesignsystems.com/color/color-models-and-color-spaces/index.html}, urldate = {2022-01-25}}

### Mathematical principles of conversion from RGB to HSV color space

Precondition: The values of R, G and B must be between 0 and 1.

Postcondition: Hue must be between 0° and 360°; Saturation and Value must be between 0 and 1.
Source: @article{gonzalez2002richard,
title={Richard E. woods},
author={Gonzalez, Rafael C},
journal={Digital image processing},
volume={2},
pages={550--570},
year={2002},
publisher={Beijing: Publishing House of Electronics industry, 20B2}
}

### Converting RGB to HSV color space in Python

### Practical application on the course example

Now that we are familiar with the conversion from RGB to HSV color space, we can apply this to our specific course example. Assuming our lane lines are blue, the RGB color space at some areas would appear in lighter and others in darker blue. In HSV color space, the Hue component will render the entire blue lane line as one color regardless of its shading. It is best to illustrate with the following image. Notice both lane lines are now roughly the same magenta color.

 Fig. 2 Rendering of entire blue color

Once the image is in HSV color space, we can seperate all the blueish colors from the image by specifying a range of the color Blue. In Hue color space, the blue color is in about 120–300 degrees range, on a 0–360 degrees scale. Note OpenCV uses a range of 0–180, instead of 0–360, so the blue range we need to specify in OpenCV is 60–150 (instead of 120–300) for example.

 Fig. 3
HSV Color Space in Degrees
@Online {FileHslhsvmodelsbsvgWikimediaCommons-2022-01-21, title = {File:Hsl-hsv models b.svg - Wikimedia Commons}, date = {2022-01-21}, year = {2022}, file = {:./references/wiki-File:Hsl-hsv_models_b.svg.html:html}, url = {https://commons.wikimedia.org/wiki/File:Hsl-hsv_models_b.svg}, urldate = {2022-01-25}}

Fig. 4 - Python code to seperate blue colors

### Masking

To "mask" out our lane lines we apply the mask operations to our image in the representation of matrices. This is quite simple. The idea is that we recalculate the value of each pixel in an image according to a mask matrix. This mask contains values that adjust the influence of neighboring pixels (and the current pixel) to the new pixel value. From a mathematical point of view, we form a weighted average with the values we set.

Let's consider the issue of an image contrast enhancement method. Basically we want to apply for every pixel of the image the following formula:

Fig. 4 Formula for masking an image

The first notation is by using a formula, while the second is a compacted version of the first by using a mask. You use the mask by putting the center of the mask matrix (in the upper case noted by the zero-zero index) on the pixel you want to calculate and sum up the pixel values multiplied with the overlapped matrix values. It's the same thing, however in case of large matrices the latter notation is a lot easier to look over.

Source: @Online {OpenCVMaskoperationsonmatrices-2022-01-24, title = {OpenCV: Mask operations on matrices}, date = {2022-01-24}, year = {2022}, file = {:./references/3.4-d7-d37-tutorial_mat_mask_operations.html.html:html}, url = {https://docs.opencv.org/3.4/d7/d37/tutorial_mat_mask_operations.html}, urldate = {2022-01-24}}

#### Python Code for Masking

Fig. 5 Masking in Python

And this is how our result looks after applying the HSV color space and successfully masking.

Fig. 6 - Blue Area Mask

## Test Yourself!

1. Since when have cars been equipped with Lane Keep Assist systems?

*Since 2018*

2. What does this system use to detect lane lines and to steer so that the car stays in the middle of the road?

*It uses a windshiled mount camera to detect lane lines.*

3. What are the components of a Lane Keep Assist System?

*- Perception (lane detection) and*
*- Path/Motion Planning (steering)*

4. What is the job of a lane keep assist system?

*The job of the lane keep assist system is to turn a video of the road into the coordinates of the detected lane lines.*

5. Name 3 popular color spaces you know.
*The most popular color spaces are RGB (red, green, blue), CMYK (cyan, magenta, yellow, black) and HSV (hue, saturation, value.*

6. What is the main idea behind turning the color space from RGB (Red/Green/Blue) into HSV (Hue/Saturation/Value)?

*The main idea behind this is that in an RGB image, different parts of a for example blue tape may be lit with different light, resulting them appears as darker blue or lighter blue. In HSV color space, the Hue component will render the entire blue tape as one color regardless of its shading. It is best to illustrate with the following image. Notice both lane lines are now roughly the same magenta color.*

7. What is the name of the computer vision package we will use in this course?

*OpenCV*

8. With how many optimized algorithms does this package come?

*With more than 2500 optimized algorithms.*

9. In figure 3 we have seen that in HSV color space the color Blue is in between a range about 120–300 degrees. What would you say in what color range is the color green?

*In a range between and 60 and 180 degrees.*

10. OpenCV uses a range of 0–180 degrees, instead of 0–360 degrees right or wrong?

*right*

11. What is the result of changing the image in HSV color space?

*Once the image is in HSV color space, we can seperate all the blueish colors from the image by specifying a range of the color Blue.*

12. What representation of our image do we need to "mask" out the lane lines?

*To “mask” out our lane lines we apply the mask operations to our image in the representation of matrices.*

13. What is the precondition of the mathematical principles of conversion from RGB to HSV color space?

*The Precondition is, that the values of R, G and B must be between 0 and 1.*

14. What is the postcondition of the mathematical principles of conversion from RGB to HSV color space?

*The postcondition is, that Hue must be between 0° and 360°; Saturation and Value must be between 0 and 1.*

15. What is the main idea of our masking-process?
*The idea is that we recalculate the value of each pixel in an image according to a mask matrix.*

16. What does the mask contain (related to neighboring and new pixel values)?
*This mask contains values that adjust the influence of neighboring pixels (and the current pixel) to the new pixel value.*
17. By masking, what do we form from a mathematical point of view?
*From a mathematical point of view, we form a weighted average with the values we set.*
18. In Fig. 4, what is the difference between the representation of the first and second natation?
*The first notation is by using a formula, while the second is a compacted version of the first by using a mask.*
19. In case of large matrices, the latter notation is easier or harder to look over?
*In case of large matrices the latter notation is a lot easier to look over.*
20. What programming language will we use to program our autonomous self-driving vehicle?
*asdfasfd*

## Important Information

For further information and additional literature please use the following sources:

Yan Jiang, Feng Gao and Guoyan Xu, "Computer vision-based multiple-lane detection on straight road and in a curve," 2010 International Conference on Image Analysis and Signal Processing, 2010, pp. 114-117, doi: 10.1109/IASP.2010.5476151.

S. K. Vishwakarma, Akash and D. S. Yadav, "Analysis of lane detection techniques using openCV," 2015 Annual IEEE India Conference (INDICON), 2015, pp. 1-4, doi: 10.1109/INDICON.2015.7443166.

M. V. G. Aziz, A. S. Prihatmanto and H. Hindersah, "Implementation of lane detection algorithm for self-driving car on toll road cipularang using Python language," 2017 4th International Conference on Electric Vehicular Technology (ICEVT), 2017, pp. 144-148, doi: 10.1109/ICEVT.2017.8323550.

S. J. Mankar, M. Demde and P. Sharma, "Design of computer vision intelligent system for lane detection," 2016 Online International Conference on Green Engineering and Technologies (IC-GET), 2016, pp. 1-3, doi: 10.1109/GET.2016.7916843.

Z. Sun, "Vision Based Lane Detection for Self-Driving Car," 2020 IEEE International Conference on Advances in Electrical Engineering and Computer Applications( AEECA), 2020, pp. 635-638, doi: 10.1109/AEECA49918.2020.9213624.

Burger, Wilhelm; Burge, Mark J.: Principles of Digital Image Processing: Fundamental Techniques. Undergraduate Topics in Computer Science. Springer-Verlag London, 1st edition, 2009.

Joblove, George H; Greenberg, Donald: Color spaces for computer graphics. In: ACM siggraph computer graphics. volume 12. ACM, pp. 20–25, 1978.