# CNN Experiments
###### tags: `experiments`

**Training Setup:**
In deep learning literature use of transfer learning has always shown better results specially when there is shortage of training data for specific domain, in our case plankton species identification/classification with highly imbalanced dataset with 39 classes, having as low as around 35 (please recheck) samples for a few classes and 40000 samples for 1 class. So We tried Resnet50 and VGG16 pretrained on ImageNet [ref]. Experimenting with Resnet50 and VGG16 we attached 1 fully connected layer after the CNN feature extractor, with number of neurons depending upon number of classes, in our case 2, 3 and 39 classes for 3 levels of plankton classification.

We used Cross Entropy Loss [ref], SGD (Stochastic gradient decent) [ref] with initial learning rate of 0.001 and 0.9 momentum. We also used a learning rate schedular StepLR [ref] with step size 7 and gamma value 0.1. We did train test split using k-fold stratification with 10% ? (confirm from Robert or Miruna).

For all of our experimentations with CNNs above settings remained the same.

**Experiments:**
We started experimenting with pretrained resnet50 [ref] +1FC , specifically we used pytorch for all of our experimentations so we used Pytorch [ref] implementation and publicly available learned parameters of resnet50 from torchvision. We first trained the model with 256x256 input image size, scaled image pixel values to range [0.0, 1.0] as float tensor(see ToTesnor from torchvision transforms). Images in dataset were of different sizes from less than 100 pixels to greater than 1000 pixels so we used bilinear interpolation to resize all images to 256x256 and trained the model for 25 epochs. During the first few epochs the model converged reaching accuracy 91% on test set, and in remaining epochs accuracy and loss stopped improving. As our next experiment we repeated previous experiment with image size 40x40 as we thought we were overkilling with such large image size but it dropped the accuracy to 89%. The we added some augmentations like Random Gray Scale with p=0.5; Random Horizontal Flip with p=0.5; Random Vertical Flip with p=0.5 as our next experiment keeping the image size 40x40 it improved the accuracy by 1% and in next experiments we increased image size to 128x128 and then again to 256x256 keeping the previous set of augmentations and it improved the accuracy (see table…).

So far the model was not improving beyond 92.8% so we tried to handle the class imbalance using Weighted Random sampler [pytorch ref]. We assigned weights to each class relative to its contribution in dataset as the low the number of samples the higher the weight (implementation is available at….). But It did not improve the results further so we stopped the experiments with resnet50.

Because of time constraints of the DSG, We could not repeat the same amount of experiments with VGG16 so only trained the VGG16 with configuration with which resnet50 preformed highest but It could not improve results beyond 91.4%.

Evaluation metrics we used are Precision, Recall, Accuracy and F1 score for the best performed model. (Tables from )

## `label2` 3 Classes
### Resnet50 +1FC
#### Exp1:
Notebook: `torch_resnet50_label2_raw`
Image Size: 256x256
Epochs: 25
Transforms:
- ToTensor
- Resize

Test Accuracy: 97%
Model: resnet50_001.pth
Confusion Matrix:

## `label3` 39 Classes
### Resnet50 +1FC
Training time for this architecture is around 1.5mins for 1 epoch.
#### Exp1
Notebook: `torch_resnet50_label3_raw`
Image Size: 256x256
Epochs: 30
Transforms:
- ToTensor
- Resize

Test Accuracy: 91%
Model: resnet50_002.pth
Confusion Matrix:

#### Exp2
Image Size: 40x40
Epochs: 25 (was Not converging after that)
Test Accuracy: 89%
#### Exp3
Image Size: 40x40
Epochs: 25
Transforms:
- ToTensor
- Resize -> 40x40
- RandomGrayScale -> 0.5
- RandomHorizontalFlip -> 0.5
- RandomVerticalFlip 0.5

Accuracy: 90%
#### Exp4
Image Size: 128x128
Epochs: 35
Transforms:
- ToTensor
- Resize
- RandomGrayScale
- RandomHorizontalFlip
- RandomVerticalFlip

Accuracy: 92.2%
Model: resnet50_003.pth
#### Exp5
Image Size: 256x256
Epochs: 25
Transforms:
- ToTensor
- Resize
- RandomGrayScale
- RandomHorizontalFlip
- RandomVerticalFlip

Accuracy: 92.8%
Model: resnet50_004.pth
#### Exp6
WeightedRandomSampler -> replacement True

Image Size: 256x256
Epochs: 25
Transforms:
- ToTensor
- Resize
- RandomGrayScale
- RandomHorizontalFlip
- RandomVerticalFlip

Accuracy: 92.1%
Model:
#### Exp7
WeightedRandomSampler -> replacement False

Image Size: 256x256
Epochs: 25
Transforms:
- ToTensor
- Resize
- RandomGrayScale
- RandomHorizontalFlip
- RandomVerticalFlip

Accuracy: 92.3%
Model:
#### Exp4 (Cancelled)

Weighted Loss and/or WeightedRandomSampler

# Classwise Results of Experiments
|Total Samples Test|Classes|Exp1|Exp2|Exp3|Exp4|Exp5|Exp6|Exp7|
|------------------|------------|---|---|---|---|---|---|---|
|14|annelida_polychaeta|0.7143|0.4286|0.5714|0.8571|0.7857|0.6429|0.7857|
|31|appendicularia|0.9677|0.9355|1|1|1|1|1|
|17|bivalvia-larvae|0.9412|0.9412|0.9412|1|1|1|1|
|9|byrozoa-larvae|0.6667|0.6667|0.8889|0.7778|0.7778|0.6667|0.7778|
|15|chaetognatha|0.7333|0.5333|0.6667|0.7333|0.7333|0.8|0.6667|
|30|cirripedia_barnacle-nauplii|0.7667|0.7667|0.9|0.9667|1|1|0.9667|
|14|cladocera|0.9286|0.8571|0.9286|1|1|1|1|
|12|cladocera_evadne-spp|0.9167|0.9167|1|1|1|1|1|
|25|cnidaria|1|0.92|1|1|1|1|1|
|166|copepod_calanoida|0.4518|0.3735|0.3735|0.488|0.488|0.4398|0.4699|
|45|copepod_calanoida_acartia-spp|0.4667|0.4667|0.5111|0.5333|0.6222|0.6222|0.5333|
|34|copepod_calanoida_calanus-spp|0.7059|0.7941|0.7941|0.7941|0.8824|0.8529|0.7941|
|4|copepod_calanoida_candacia-spp|0.5|0.5|0.25|0.75|0.5|0.5|0.5|
|77|copepod_calanoida_centropages-spp|0.6753|0.4935|0.5065|0.7662|0.7922|0.7662|0.8052|
|198|copepod_calanoida_para-pseudocalanus-spp|0.7071|0.5707|0.6869|0.7626|0.7626|0.8232|0.7879|
|16|copepod_calanoida_temora-spp|0.3125|0.25|0.3125|0.5625|0.5625|0.5625|0.5625|
|10|copepod_cyclopoida|0.2|0|0|0|0.1|0|0|
|111|copepod_cyclopoida_corycaeus-spp|0.8739|0.8559|0.8468|0.8829|0.8829|0.8919|0.8649|
|49|copepod_cyclopoida_oithona-spp|0.6327|0.6531|0.7347|0.6735|0.7143|0.6939|0.6735|
|71|copepod_cyclopoida_oncaea-spp|0.662|0.5211|0.5915|0.7042|0.7465|0.7324|0.7465|
|64|copepod_harpacticoida|0.6719|0.5625|0.6719|0.7031|0.6875|0.625|0.7031|
|138|copepod_nauplii|0.7971|0.7754|0.7826|0.8116|0.8406|0.8116|0.8333|
|185|copepod_unknown|0.4324|0.4324|0.3892|0.4378|0.4324|0.3892|0.4108|
|4|decapoda-larvae_brachyura|0.25|0|0.25|0|0|0.5|0.25|
|4000|detritus|0.9933|0.9898|0.9883|0.9927|0.996|0.9915|0.9935|
|79|echniodermata-larvae|0.9367|0.9114|0.9241|0.9494|0.9367|0.9494|0.9494|
|16|euphausiid|1|0.9375|1|1|1|1|1|
|13|euphausiid_nauplii|0.4615|0.0769|0.1538|0.3846|0.6923|0.4615|0.4615|
|20|fish-eggs|0.95|0.95|1|1|1|1|1|
|16|gastropoda-larva|0.8125|0.5|0.875|1|1|1|0.875|
|13|mysideacea|1|0.6923|0.9231|1|1|1|1|
|35|nt-bubbles|1|1|1|1|1|1|1|
|45|nt-phyto_ceratium-spp|0.9778|0.9556|0.9556|0.9778|0.9778|0.9333|0.9556|
|18|nt-phyto_rhizosolenia-spp|1|0.8889|0.8333|0.9444|0.9444|1|1|
|29|nt_phyto_chains|0.8966|1|1|1|1|0.931|1|
|18|ostracoda|0.8333|0.4444|0.8889|1|1|1|1|
|181|radiolaria|0.9945|0.9945|0.989|0.9945|0.9945|0.9945|0.9945|
|12|tintinnida|0.9167|0.9167|1|1|1|1|1|
|29|tunicata_doliolida|1|1|1|1|1|1|1|

### VGG16 +1FC
#### Exp1
Image Size: 256x256
Epochs: 25
Transforms:
- ToTensor
- Resize
- RandomGrayScale
- RandomHorizontalFlip
- RandomVerticalFlip

Accuracy 91.4%
Model: vgg16_001

|Total Samples in Test|Classes|Exp1|
|----------------------|------|-----|
|14|annelida_polychaeta|0.5|
|31|appendicularia|1|
|17|bivalvia-larvae|1|
|9|byrozoa-larvae|0.7778|
|15|chaetognatha|0.8|
|30|cirripedia_barnacle-nauplii|0.9667|
|14|cladocera|0.9286|
|12|cladocera_evadne-spp|0.9167|
|25|cnidaria|1|
|166|copepod_calanoida|0.3675|
|45|copepod_calanoida_acartia-spp|0.5556|
|34|copepod_calanoida_calanus-spp|0.8529|
|4|copepod_calanoida_candacia-spp|0.75|
|77|copepod_calanoida_centropages-spp|0.7792|
|198|copepod_calanoida_para-pseudocalanus-spp|0.7576|
|16|copepod_calanoida_temora-spp|0.5|
|10|copepod_cyclopoida|0.1|
|111|copepod_cyclopoida_corycaeus-spp|0.8649|
|49|copepod_cyclopoida_oithona-spp|0.5714|
|71|copepod_cyclopoida_oncaea-spp|0.7465|
|64|copepod_harpacticoida|0.6562|
|138|copepod_nauplii|0.8116|
|185|copepod_unknown|0.4216|
|4|decapoda-larvae_brachyura|0|
|4000|detritus|0.9905|
|79|echniodermata-larvae|0.9494|
|16|euphausiid|1|
|13|euphausiid_nauplii|0.3846|
|20|fish-eggs|1|
|16|gastropoda-larva|0.875|
|13|mysideacea|1|
|35|nt-bubbles|1|
|45|nt-phyto_ceratium-spp|0.9556|
|18|nt-phyto_rhizosolenia-spp|0.9444|
|29|nt_phyto_chains|0.931|
|18|ostracoda|0.9444|
|181|radiolaria|0.9945|
|12|tintinnida|1|
|29|tunicata_doliolida|1|