# The Joy of Testing

Todays blog post is going to be a little bit different, we are going to be documenting the testing we have done during the devolopment of this project. The post will be a little bit dry compared to the others and we recommened that you consider skipping it, unless you are really interested in the different results from our various attempts at getting our system working.

As we've outlined before, an ANPR system follow some usual steps. We have broken our testing down into the following headings based on these steps;

- Plate Extraction
- Character Segmentation
- Character Recognition
- Full System

*****
## Plate Extraction

### Plate Extraction without using NN
Our first attempt at plate extraction was based on Connected Component labeling, and we go into it in more depth in the blog post [here.](https://frosty-goldberg-5fed5e.netlify.com/posts/plate-extraction/).

| | Test 1 | Test 2 |
| -------- | -------- | -------- |
| Images Tested | 10 | 50 |
| Results | 80% | 25% |

#### Comments:
For Test 1 we considered it a success if a correct bounding box for a license plate was found, regardless if other bounding boxes where found which where incorrect. For Test 2 we considered it a success if the only bounding boxes found where around a license plate.

The first test performed much better as the test images we had where almost entirely of Irish License plates, and where taken from mostly shallow angles. For the second test we used more varied images, which were more representive of images which will actually be used by our system.

#### Conclusion:
From these tests we found that this method of plate extraction would not work for our uses, however we did make great use of this program to build a dataset of images for use later in our project.

### Plate Extraction with NN

| | Test 1 | Test 2 | Test 3 |
| -------- | -------- | -------- | ------ |
| Images Tested | 50 | 50 | 300 |
| Results | 50% | 85% | 95% |
| Model Used | 1 | 2 | 2 |
| Minimun % Probability | 50% | 35% | 33%

#### Sample Result Image
#### Comments:
Test 1 used our first model generated, this was created after about 16 hours of training on a CPU. We tested at this point, while the loss was still fairly high, in order to check the progress of the training. This was done as we were still quite new to NN training and were sure when to stop training to avoid overfitting.

Test 2 and 3 used our second model taken from training which seemed to perform the best of all the models we had generated. The minimun percentage probabilty is the lowest confidence level that the system will still consider correct. We found that 33% gave the best results, while minimizing false positives.
#### Conclusion:
After the second and the third test we were quite happy with the results, and felt that we could proceed to the next stage of our project, marking plate extraction as complete.

*****
## Character Segmentation
### Character Segmentation without using NN

| | Test 1 | Test 2 |
| -------- | -------- | -------- |
| Images Tested | 10 | 100 |
| Results | 90% | 25% |

#### Comments:
Our first attempt at character segmentation was similar to our first plate extraction attempt, using connected component labeling. See [this](https://frosty-goldberg-5fed5e.netlify.com/posts/character-segmentation/) blog post for more information on it. Our first test performed well as the test images were all "ideal" images for this system. They were clear photos taken directly in front of the license plate. The second test we considered a success if the it clearly segmented at least 75% of the characters in a given license plate.

#### Conclusion:
The results of these tests were expected, and turned out very similar, although slightly better, than the tests on our original plate extraction program. Overall it performed quite well on images taken from a shallow angle, but fell down when the angle was steeper and failed completely when there was overlap between character.

From the above results we knew that this program would work for our system. We still used this program to generate some images for training and testing our character recognition program, so we were glad we spent the time developing it.

### Character Segmentation using NN

| | Test 1 |
| -------- | -------- |
| Images Tested | 300 |
| Results | 50% |
| Model Used | 1 |

#### Sample Result Image
#### Comments:
The minimun percentage probability was left at 35% for all of the above tests. This results differed from previous tests as the success rate didn't seem to be effected by the angle the images were taken from. This was one of the benefits of using neural nets trained on a variety of images.

Unlike the previous character segmentation test, we only considered these a success if they corrected segmented 85% of the characters, as this did not struggle with steep angles or overlapping characters.

#### Conclusion:
Overall the results for this program were quite good, and we felt that with increasing the size of the dataset and more training would return an excellent increase in results. We eventually decided to not go ahead with this however as the output of this program was to be fed to the k-NN OCR, which we had to decided to not devolop further at this point.

*****
## Character Recognition

### Character Recognition using k-NN

| | Test 1 | Test 2 | Test 3 |
| -------- | -------- | -------- | ------ |
| Images Tested | 50 | 50 | 100 |
| Results | 40% | 70% | 50% |
| Model Used | 1 | 2 | 3 |
#### Comments:
For model 1 and 2, these only work with numbers, and not with letters, as such they were test only on numbers. The test images they were tested on were not representive of our expected data as they were mostly clear and from a shallow angle. The third model and test were on all characters, both numbers and letters, and the test data better represented the type of images we planned to work with.

#### Conclusion:
Overall the k-NN OCR worked well, but struggled with images which were taken from an angle or were of poor quality. This was supported by other systems we found online which were only shown working with ideal images. This OCR did work very well with clear images though, so while it may not work for our system it would be quite useful for others, such as digitizing scanned documents.

### Character Recognition using NN

| | Test 1 | Test 2 |
| -------- | -------- | -------- |
| Images Tested | 10 | 100 |
| Results | 10% | 17% |
| Model Used | 1 | 2 |
|Minimum % Probability | 50% | 95% |

#### Sample Result Images
 
#### Comments:
Model 2 had been trained for longer than model 1, on the same images. This was done to bring the loss down to an acceptable level.

It was difficult to assess whether a particular image had it characters successfully recognized. This was because our NN was recognizing a huge number characters, and multiple successful recognitions for each characters. This continued even after setting the minimum percentage probability to 95%. Despite all this, it was still recognizing many of the characters successfully.

#### Conclusion:
From these results it was clear to us that the net simply didn't have even data to train on. For more information on this please see [this](https://frosty-goldberg-5fed5e.netlify.com/posts/creating-an-ocr-engine-part-2/) blog post. It was clear the amount of data we would need to generate was out of the scope of our project, so we looked at using alternatives.
### Character Recognition using Google Vision

| | Test 1 |
| -------- | -------- |
| Images Tested | 300 |
| Results | 97% |
#### Comments:
We set up google vision and used the same test images that we used on our previous OCR tests. One issue encountered was that the google vision OCR would pick up some very small characters on license plates and their surrounds, however we didn't could this as a failure if it still correctly picked up the rest of the plate.
#### Conclusion:
We were very happy with these results, and decided to move forward with using google vision as our OCR engine. We would have liked to do more tests but we were limited with how much we could use google vision before we would incur charges.
*****
## Complete System

### Complete System on Pythonanywhere

| | Test 1 | Test 2 |
| -------- | -------- | ------ |
| Images Tested | 50 | 50 |
| Average RunTime|17 Seconds|10 seconds|
#### Comments:
This test was timed from when the upload button was clicked, until the the result was displayed on the webpage. All tests were done on the site while it was live, rather than locally. Test 2 was performed after optimizations were made.
#### Conclusion:
Even after the optimizations were made, we were still far off our target time of <5 seconds. Locally we had it performing much faster with the use of multi-threading, which is not supported with Pythonanywhere, so we decided to lost at other hosting options.

### Complete System on Ngrok

| | Test 1 |
| -------- | -------- |
| Images Tested | 50 |
| Average RunTime| 3 Seconds|
#### Comments:
This test was also time from when the upload button was clicked, until the result was displayed. This was done while the site was live, and from a different machine, on a different LAN, than the one hosting the website.

#### Conclusion:
Using Ngrok to host on our own machines allowed us to implement our optimizations using multi-threading. This resulted in much better performance and we were far under our 5 second goal. We decided that we were happy with these results, and would use Ngrok as our hosting solution.