# Brainstorming
## Random ideas that pop into your head while looking at the data
###### tags: `brainstorm`

Ideas/comments:

* [Sari] First separate level 1. Then look into identifying level 3?
* [Sari] Plot spatial distribution of organisms to see whether there are spatial trends that could be utilized for classification.
* Need csv file wil lat, long, and level 1-3 for each image. Anyone can do that quickly?
* [Sari] Size likely critical for identification. Is current padding sensible? Are there other ideas?
* [Sari] Larger images are typically more important as detritus is typically smaller and not so interesting (for this purpose anyway)

* [Jen] Synthetically generate lower class samples to improve class imbalance
* GANs
* [Jen] Explore additional pre-processing techniques that can enhance differences between classes (esp. detritus v. plankton)
* E.g. resizing images
* [Jen] Pipeline of binary classifiers (e.g. step 1: is it detritus? step 2: is it zooplankton? step 3: further class specification)
* This could help break up our work so people could focus on parts of the pipeline

* [Charlie] To help with an initial exploration of the data, and to help illustrate the clusters in the report - once we have got the x/y / long/lat data from the images, we can visualise clusters of different classes - to visualise the spatial patterns of the pre-classified/identified data and then look at how this compares to the spatial patterns of the data that we classify.

Explainable API
* Lower level feature exploration (e.g. edges, color, SIFT, HoG) for explainable features of different classes

* Staged classifiers

* Use prediction scores

* Data augmentation. Can change size a little bit (maybe +/- 5%) as this would reflect natural interspecies variabily.

* Data augmentation. Can we introduce noise in colour channels?

* [Charlie] It could be useful just for illustration purposes/perhaps as a side-track to visualise the spatial patterns of the pre-classified/identified data and then look at how this compares to the spatial patterns of the data that we classify.

* [Rob] should we be using cross fold validation so that we get error bars on our classifiers?

* We probably will need GPU compute - can we document the training time for each experiment?

* Mixed model inputs: https://www.pyimagesearch.com/2019/02/04/keras-multiple-inputs-and-mixed-data/

## Potential hypotheses

Color versus monochrome, impact on classification.

Image only vs image + metadata, impact on classification.

Which features of an image are most discrimantory for our classifers?

## Potential results figures

Colour versus monochrome - side by side comparison

A map with spatial distribution of the data. [prediction vs observed]

## Comments on State of the Art
**EcoTaxa**: Most work carried out on black-and-white plankton images on EcoTaxa [https://ecotaxa.obs-vlfr.fr/]. Uses random forest on lower level features. However, it now as a deep learning model as well, though it's less frequently used.
EcoTaxa is only intended as a classification support - the image classification is predicted. Images are then groups according to prediction and sorted accoring to prediction score. A human then has to verify each images individually.

**Colour**: Most plankton images are black-and-white (see EcoTaxa), and EcoTaxa actually converts colour images into black-and-white. It has been suggested that inclusion of colour massively increases classification accuracy.
*"Inclusion of color has been shown to produce much higher classification accuracy with automated routines than monochrome images (Saminsky and Gallager, 2018)."*
**Saminsky, M. W., and Gallager, S. M** (2018). “Plankton image classification, storage, visualization, and analytics leveraging cloud computing,” in Poster Presentation at Ocean Sciences Meeting 2018 (Portland).

**Add metadata**:
"Improving plankton image classification using context metadata"
https://aslopubs.onlinelibrary.wiley.com/doi/full/10.1002/lom3.10324

*We boost the performance of CNN classifiers by incorporating metadata of different types and illustrate how to assimilate metadata beyond simple concatenation. We utilize both geotemporal (e.g., sample depth, location, time of day) and hydrographic (e.g., temperature, salinity, chlorophyll a) metadata and show that either type by itself, or both combined, can substantially reduce error rates.*

### Plankton Imager
(Scott et al. 2021) [In situ automated imaging, using the Plankton Imager, captures temporal variations in mesozooplankton using the Celtic Sea as a case study](https://academic.oup.com/plankt/article-abstract/43/2/300/6178430?redirectedFrom=fulltext)

# Review / intro parts (to go into Overleaf)

1. General intro to problem (Sari)
* Plankton - why it matters, why we need images and AI/ML.

2. Augmentation (Meghna, Jennifer)
* Table of possible techniques with refs?
* Advantages / disadvantages
* Concluding --> what we going to use (first)

3. Model (Meghna, Miruna?)
* Short table with models
* Short lit rev for models used in plankton
* Concluding --> what we going to use (first)

# Some links shared by Evangeline (PI)
* https://stanford.edu/~shervine/teaching/cs-230/cheatsheet-convolutional-neural-networks#overview
* https://www.kaggle.com/pavan1512/a-guide-to-choose-best-cnn-architecture
* https://towardsdatascience.com/a-comprehensive-hands-on-guide-to-transfer-learning-with-real-world-applications-in-deep-learning-212bf3b2f27a?gi=818b79349880
* https://www.tensorflow.org/tutorials/images/transfer_learning

# Dec 7 CNN Brainstorming:
* Image loading, pre-processing, lower level feature explore (Jen)
* Create new folder of augmented data
* Tensorflow augmentation (Rohit)
* Does it save images or just feed it into training?
* GANs augmentation (Jen)
* CycleGAN (do we have enough classes)
* GPU setup (Meghna)
* State of the Art Models (Meghna, Attiq)