# HEIG-VD MLG Pratical Work 5
Authors: **Bécaud Arthur**, **Egremy Bruno**

## SOM Part 1
### Exercise 1: Animals database clustering with Kohonen Self-Organizing Maps

<p style="background-color:#006600; color:#fff;padding:5px; font-weight:bold">Q: Analyze the code (in KohonenUtils.py) to understand what are the uses of each block in the cell right above.</p>

The first block creates a Kohonen Self Organizing Map with the following parameters in order: number of lines, number of columns, and the number of animal's attributes.

The second choose the learning parameters: number of iteration, learning rate, and neighborhood size.

The third trains the model.

And the last one makes the U-Matrix plot.

<p style="background-color:#006600; color:#fff;padding:5px; font-weight:bold">Q: What do you think about the clustering quality ? </p>

There is consistent clustering correctly separating, first, birds from mammals, and then each subspecies profile of these groups.

For example, horses and zebras are clustered together, while lions, tigers, and cats are close to each other.

<p style="background-color:#006600; color:#fff;padding:5px; font-weight:bold">Q: We plotted the U-Matrix in the previous cell. What does it represent ? </p>

The U matrix shows the distance between each species. The closer it is, the more similar the characteristics of the species are.

<p style="background-color:#006600; color:#fff;padding:5px; font-weight:bold">Q: What do the small stars represent ? What do the bigger circles represent ? </p>

The small stars/points represent the neurons, while the bigger circles represent the distance between the neurons.

<p style="background-color:#006600; color:#fff;padding:5px; font-weight:bold">Q: What does the color in the big circles mean ? </p>

The darker the color, the greater the distance between species. The lighter the color, the more similar the species are.

<p style="background-color:#006600; color:#fff;padding:5px; font-weight:bold">Q: What does the color in the small stars mean ? </p>

The red stars represent one to many animals. The blue stars are unused neurons.

<p style="background-color:#006600; color:#fff;padding:5px; font-weight:bold">Q: Plot the learning rate and the neighborhood size. Why do you think we choose them like this? </p>


The learning rate and the neighborhood size must have been chosen like this to make them follow along. It allows the model training to separate more efficiently the animals as the iteration progress.

### Exercise 2: Animals database clustering with K-Means

<p style="background-color:#006600; color:#fff;padding:5px; font-weight:bold">Q: Observe the animals that are grouped together by K-Means and try different numbers of clusters: K=2,3,4, etc. </p>

K = 2: birds are separated from mammals.

K = 3: birds are separated from mammals, and a second separation tends to occur within birds or mammals, such as the group [fox, dog, wolf] from other mammals, or the group [duck, goose] from other birds. This third group is chosen randomly from the two cited previously.

K > 4: the larger the K, the more separation there is within the groups of birds or mammals. The supplementary groups are chosen randomly.

<p style="background-color:#006600; color:#fff;padding:5px; font-weight:bold">Q: n_init is a parameter that automatically asks K-means to try different cluster initializations and selects the best result. init='random' asks K-means to randomly initialize the cluster centroids. Please, try init=’k-means++’ and modify n_init to 10 for example and observe the results. </p>

The results are very similar to the previous ones, but the randomness is gone, giving a stable result where the separated groups tend to be always the same.

### Exercise 3 : Wine database clustering with K-Means

<p style="background-color:#006600; color:#fff;padding:5px; font-weight:bold">Q: Observe the observations that are grouped together by K-Means. </p>

The expected clusterization of features is obtained as follows :

from sklearn.datasets import load_wine

data = load_wine()
features = data.data
classes = data.target

print(classes)
# [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
#  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
#  1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
#  1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
#  2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2]

The result of the clusterisation of the features :

kmeans = KMeans(n_clusters=3,n_init=10, init='k-means++').fit(features)
print(kmeans.labels_)
# [0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 1 1 0 0 1 0 0 0 0 0 0 1 1
#  0 0 1 1 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 1 2 1 2 2 1 2 2 1 1 1 2 2 0
#  1 2 2 2 1 2 2 1 1 2 2 2 2 2 1 1 2 2 2 2 2 1 1 2 1 2 1 2 2 2 1 2 2 2 2 1 2
#  2 1 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 1 2 2 1 1 1 1 2 2 2 1 1 2 2 1 1 2 1
#  1 2 2 2 2 1 1 1 2 1 1 1 2 1 2 1 1 2 1 1 1 1 2 2 1 1 1 1 1 2]

The result mainly shows the difficulty for KMeans in clustering classes 1 and 2. Class 0 is globally correctly clustered.

<p style="background-color:#006600; color:#fff;padding:5px; font-weight:bold">Q: Count the number of "bottles" that are correctly grouped. What is the accuracy of this unsupervised classification? </p>

A total of 85 bottles were correctly grouped, for a total accuracy of 47.75%.

<p style="background-color:#006600; color:#fff;padding:5px; font-weight:bold">Q: Try to improve the performance of the classification. Does normalizing the data increases the accuracy? Does selecting a reduced number of features improves the accuracy? Why?. </p>

The normalization of the dataset resulted in the following clusterization.

# [0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 2 2 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0
#  0 0 2 2 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 2 1 2 1 1 2 2 2 2 1 0 1 2 2
#  0 1 1 1 2 1 1 0 2 2 2 1 2 2 0 0 2 2 2 1 1 2 1 2 2 1 2 2 1 1 2 1 2 2 1 2 2
#  1 2 1 1 1 2 1 1 2 2 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 0 2 2 2 2 0 2 2 2 0 2 2
#  2 1 1 1 1 2 2 2 2 0 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2]

Making a total of 124 bottles correctly grouped, and an accuracy of 69.66%.

For the reduction of features, a features selection is made to select the `N` best features.
The features are scored as follows using `sklearn.feature_selection.SelectKBest` :


A first selection was made with the 7 best features, which resulted in the following clusterization:

# [0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 2 2 0 0 2 2 0 0 2 0 0 0 0 0 0 2 2
#  0 0 2 2 0 0 2 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 2 1 2 1 1 2 1 1 2 2 2 1 1 0
#  2 1 1 1 2 1 1 2 2 1 1 1 1 1 2 2 1 1 1 1 1 2 2 1 2 1 2 1 1 1 2 1 1 1 1 2 1
#  1 2 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 2 1 1 2 2 2 2 1 1 1 2 2 1 1 2 2 1 2
#  2 1 1 1 1 2 2 2 1 2 2 2 1 2 1 2 2 1 2 2 2 2 1 1 2 2 2 2 2 1]

Making a total of 125 bottles correctly grouped, and an accuracy of 70.22%.

The second selection was made with the 3 best features, which ended up having the same result as the 7 best features case.

# [0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 2 2 0 0 2 2 0 0 2 0 0 0 0 0 0 2 2
#  0 0 2 2 0 0 2 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 2 1 2 1 1 2 1 1 2 2 2 1 1 0
#  2 1 1 1 2 1 1 2 2 1 1 1 1 1 2 2 1 1 1 1 1 2 2 1 2 1 2 1 1 1 2 1 1 1 1 2 1
#  1 2 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 2 1 1 2 2 2 2 1 1 1 2 2 1 1 2 2 1 2
#  2 1 1 1 1 2 2 2 1 2 2 2 1 2 1 2 2 1 2 2 2 2 1 1 2 2 2 2 2 1]

To conclude, having the dataset normalized improved the result by ~20%. This is due to the improvement of the dataset through the normalizing procedure resulting in the use of more meaningful features or noise reduction within the data.

## SOM Part 2
### Extraction methods
The three feature extraction methods are part of the input preprocessing procedure. They take the images as input and produce certain features from the images.

The **extract_histogram** method runs the images through a grayscale filter and then extracts the grayscale into 10 features.

The **extract_hue_histogram** method transforms the images from RGB format to HSV format and then extracts 10 features from the HSV spectrum.

The **extract_color_histogram** method extracts 30 features from the RGB values of the images.

### Dinosaurs, Elephants and horses

The first experiment uses the dinosaurs, elephants, and horses images with the extract_histogram method.

We obtained the following result :

<center>



</center>

The result is pretty good, there is an overall good separation between the species, even if there some artifacts with the horses and elephants.

### Beaches, Monuments, and Mountains

The second experiment uses the beaches, monuments, and mountains images with the extract_histogram method.

We obtain the following result :

<center>



</center>

As the result shows, the clusterization is a mess. Almost all the grey circles inside the U-Matrix are dark meaning that almost all the neurons are far from each other. This tells us that the KMeans did not found any meaningful characteristics with the dataset for its classification.

### Flowers and food
The second experiment uses the flowers, and food images with all the methods.

#### extract_histogram method

We obtain the following result :

<center>



</center>

The separation between the flowers and the food is a bit fuzzy, as shown with the U-Matrix, and a few artifacts occurred. This first method seems to be good enough for the problem.

#### extract_hue_histogram method

We obtain the following result :

<center>



</center>

The result shows a good clusterization for the colors of the images and an overall bad separation between the flowers and the food. This second method with the RGB color is not good enough for a proper clusterization of this problem.

#### extract_color_histogram method

We obtain the following result :

<center>



</center>

The last result with the third method shows an interesting clusterization of the problem. First, the flowers and the food are properly classified, except for one flower. Then there is a good separation with the colors of the flowers with the white-yellow flowers on the left side, and the red-violet-orange on the right side. A similar. but less distinguishable, behavior happens with the food where the food with white accent is contained on the left side.