<style> .reveal section img { background:none; border:none; box-shadow:none; } </style> # DiSCo ## Dictionary-based Spectral unmixing for Coastal environmental monitoring PANAMA Thomas Cusson Jérémy Cohen --- ## Hyperspectral Imaging - Light<!-- .element: class="fragment" data-fragment-index="1" --> - Imaging<!-- .element: class="fragment" data-fragment-index="2" --> - Spectroscopy<!-- .element: class="fragment" data-fragment-index="3" --> - Hyperspectral Imaging<!-- .element: class="fragment" data-fragment-index="4" --> ---- ### Light  $\lambda = \frac{c}{\nu}$<!-- .element: class="fragment" data-fragment-index="2" --> ---- ### Imaging Number of Channels :<!-- .element: class="fragment" data-fragment-index="1" --> - One : Black and white<!-- .element: class="fragment" data-fragment-index="2" --> - Three : Red, Green and Blue<!-- .element: class="fragment" data-fragment-index="3" --> - More : Display individually, or conversion function<!-- .element: class="fragment" data-fragment-index="4" --> ---- ### Spectroscopy Analysis of spectrum<!-- .element: class="fragment" data-fragment-index="1" --> <!-- .element: class="fragment" data-fragment-index="1" --> ---- ## Hyperspectral Imaging  ---- ## Hyperspectral Imaging  ---- ## Hyperspectral Imaging ### Sources of spectral variance - The sensor - The light source - The atmosphere - Indirect lighting - Reflections - The material itself - Shadows - Everything else… --- ## Goal - Good Clustering/Segmentation<!-- .element: class="fragment" data-fragment-index="1" --> - No prior knowledge : Unsupervised<!-- .element: class="fragment" data-fragment-index="2" --> ---- ### Urban  ---- ### Urban  ---- ### Naive clustering  ---- ### Clustering problems  ---- ### Clustering problems  ---- ### Clustering problems  --- ### Spectral Unmixing Each pixel is a mixture of materials/endmembers Working assumptions<!-- .element: class="fragment" data-fragment-index="1" --> - Spectra are non-negative<!-- .element: class="fragment" data-fragment-index="2" --> - Abundances are non-negative<!-- .element: class="fragment" data-fragment-index="3" --> - Spectra combine linearly<!-- .element: class="fragment" data-fragment-index="4" --> ---- ### Spectral Unmixing #### Non-negative Matrix Factorization<!-- .element: class="fragment" data-fragment-index="1" --> Data (Known):<!-- .element: class="fragment" data-fragment-index="2" --> $X\in \mathbb{R}_+^{n\times m}$<!-- .element: class="fragment" data-fragment-index="2" --> Abundances (Unknown):<!-- .element: class="fragment" data-fragment-index="3" --> $H\in \mathbb{R}_+^{n\times r}$<!-- .element: class="fragment" data-fragment-index="3" --> Spectra (Unknown):<!-- .element: class="fragment" data-fragment-index="4" --> $W\in \mathbb{R}_+^{r\times m}$<!-- .element: class="fragment" data-fragment-index="4" --> $X_{ji}= \sum_{k=0}^r{H_{jk} W_{ki} + \epsilon}$<!-- .element: class="fragment" data-fragment-index="5" --> $X=HW +\epsilon$<!-- .element: class="fragment" data-fragment-index="6" --> ---- #### NMF  ---- ### Spectral Unmixing  ---- #### NMF  ---- ### Spectral Unmixing ### Pseudo-code for NMF 1. Initialize<!-- .element: class="fragment" data-fragment-index="1" --> 2. Optimize $H$ by Non-Negative Least Squares<!-- .element: class="fragment" data-fragment-index="2" --> 3. Optimize $W$ by Non-Negative Least Squares<!-- .element: class="fragment" data-fragment-index="3" --> 4. Go back to 2 until stopping conditions are met<!-- .element: class="fragment" data-fragment-index="4" --> --- ### The Conversion Problem Simply take biggest abundance $H_j$ for a pixel ?<!-- .element: class="fragment" data-fragment-index="1" --> Except it's not necessarily representative<!-- .element: class="fragment" data-fragment-index="2" --> $X_{ji}= \sum_{k=0}^r{\alpha H_{jk} \frac{1}{\alpha} W_{ki} + \epsilon}$<!-- .element: class="fragment" data-fragment-index="3" --> ---- ### The Conversion Problem Normalize $W$ ?  ---- ### The Conversion Problem Divide each abundance map $H_k$ by max value ?<!-- .element: class="fragment" data-fragment-index="1" --> <!-- .element: class="fragment" data-fragment-index="2" --> ---- ### Results Labeling error (%) | Method | r=4 | r=5 | r=6 | |--------------|-------|-------|-------| | Ground-truth | 0 | 0 | 0 | | Kmeans | 64.96 | 48.69 | 50.42 | | NKmeans | 7.36 | 29.76 | 34.02 | | best NMF | 12.92 | 15.03 | 32.83 | <!--- ### Results Average reconstruction error (%) | Method | r=4 | r=5 | r=6 | |--------------|-------|-------|-------| | Ground-truth | 7.59 | 8.82 | 8.53 | | Kmeans | 31.22 | 28.34 | 26.02 | | NKmeans | 11.87 | 10.42 | 9.72 | | NMF | 4.34 | 3.79 | 3.33 | --> --- ## Goal - Good Clustering/Segmentation - No prior knowledge : Unsupervised - Prior knowledge : Semi-supervised<!-- .element: class="fragment" data-fragment-index="1" --> ---- ## Semi-supervised - Dataset<!-- .element: class="fragment" data-fragment-index="1" --> - Clustering supervised by Dictionary spectra<!-- .element: class="fragment" data-fragment-index="2" --> - Blind NMF results projected on Dictionary<!-- .element: class="fragment" data-fragment-index="3" --> - Voting system<!-- .element: class="fragment" data-fragment-index="4" --> ---- ### The data : Hyperspectral image  ---- ### The data : Spectra library - 14 Materials<!-- .element: class="fragment" data-fragment-index="1" --> - From 5 to 40 spectra for each<!-- .element: class="fragment" data-fragment-index="2" --> - Total of 200 spectra<!-- .element: class="fragment" data-fragment-index="3" --> - Spectra from 287 to 1094nm, 512 samples<!-- .element: class="fragment" data-fragment-index="4" --> ---- ### The data : Anotated patches  ---- ### Naive supervised clustering using Dictionary Simply take the largest similarity !<!-- .element: class="fragment" data-fragment-index="1" --> Normalize vectors :<!-- .element: class="fragment" data-fragment-index="2" --> $\vec{u_n} = \frac{\vec{u}}{||\vec{u}||_F^2}$ and $\vec{v_n} = \frac{\vec{v}}{||\vec{v}||_F^2}$<!-- .element: class="fragment" data-fragment-index="3" --> $\rho_{\vec{u},\vec{v}} = \vec{u_n}\cdot\vec{v_n}$<!-- .element: class="fragment" data-fragment-index="4" --> ---- ### Clustering supervised by Dictionary  Baseline ---- ### Comparing patches and Dictionary <!-- .element: class="fragment" data-fragment-index="1" --> <!-- .element: class="fragment" data-fragment-index="2" --> ---- ### Spectra image/library #### Zoom on Ascophyllum  <!-- .element: class="fragment" data-fragment-index="1" --> <!--- Enlevable ? Un peu hors-sujet ### Clustering supervised by patches  Enlevable ? Un peu hors-sujet ### Clustering from abundance derived from patches spectra  --> --- ## Blind NMF then dictionary comparison ----  ----  ----  ---- ### Lookalikes  --- ### Voting system Still work in progress<!-- .element: class="fragment" data-fragment-index="1" --> $Score_{m, j}=\sum_{k=0}^r H_{k, j}\frac{\sum \rho_{W_k, D_m^\top}}{||D_m||}$<!-- .element: class="fragment" data-fragment-index="2" --> Pixel's material : Best score<!-- .element: class="fragment" data-fragment-index="3" --> - Each endmembers gives a correlation score with every library spectrum<!-- .element: class="fragment" data-fragment-index="4" --> - Each correlation with library spectrum is averaged for each material<!-- .element: class="fragment" data-fragment-index="5" --> - Each abundance gives a weight for correlations<!-- .element: class="fragment" data-fragment-index="6" --> - Final score for each material is the sum of abundances times correlations scores<!-- .element: class="fragment" data-fragment-index="7" --> --- # Conclusion Encouraging result to improve clustering with spectral unmixing.<!-- .element: class="fragment" data-fragment-index="1" --> Further testing with other techniques needed.<!-- .element: class="fragment" data-fragment-index="2" --> ---- # My contribution - Experiments with sparsity constraints<!-- .element: class="fragment" data-fragment-index="1" --> - Conversion method Unsupervised NMF→clustering<!-- .element: class="fragment" data-fragment-index="2" --> - Supervised by patch<!-- .element: class="fragment" data-fragment-index="3" --> - Voting system for clustering (WIP)<!-- .element: class="fragment" data-fragment-index="4" --> ---- # Questions