<!-- Docs for making Markdown slide deck on HackMD using Revealjs https://hackmd.io/s/how-to-create-slide-deck https://revealjs.com --> ### :airplane_departure:+:satellite:+:computer: = :snow_capped_mountain: :flag-aq: #### **DeepBedMap: A Super-Resolution GAN for resolving the subglacial topography of Antarctica** <small> Machine Learning for Polar Regions Workshop presentation at Lamont-Doherty Earth Observatory (virtual), Friday 17 Jun 2022, 15:15-15:30 (UTC) </small> _by **[Wei Ji Leong](https://github.com/weiji14)** & [Huw Horgan](https://orcid.org/0000-0002-4836-0078)_ <!-- Put the link to this slide here so people can follow --> <small> P.S. Slides are at https://hackmd.io/@weiji14/2022ML4Polar</small> --- ### Why do we need high resolution images? Going from a 1km resolution BEDMAP2 to a higher (250m) spatial resolution **bed topography** would enable us to: <span class="fragment fade-in"> - [x] Capture more glaciological processes in **ice sheet models** - Ice flows slower over rough beds compared to smooth beds </span> <span class="fragment fade-in"> - [x] Better understand Antarctica's **subglacial hydrology** - Water under the ice can lower friction and lead to faster flow </span> <span class="fragment fade-in"> Cumulatively, this will lead to more accurate **sea level rise** :ocean: projections! </span> --- ### How to get a better BEDMAP? ---- ### Direct approach - **Ice-penetrating radar** surveys, accurate but **limited spatial coverage** - E.g. from British Antarctic Survey, Operation Icebridge field missions, etc <img src="https://user-images.githubusercontent.com/23487320/49407666-392e9080-f7be-11e8-9788-061dc5040796.png" alt="Map of radio-echo-sounding datasets around Antarctica used in Gardner et al., 2018 paper" style="margin:0px auto;display:block" width="35%"/> <small>Figure showing Radio-echo-sounding datasets around Antarctica from [Gardner et al. 2018](https://doi.org/10.5194/tc-12-521-2018)</small> ---- ### Indirect approach - **Inverse** model on satellite captured **ice surface** data, less accurate but **widely applicable** - E.g. Our [**DeepBedMap**](https://doi.org/10.5194/tc-2020-74) model, [BedMachine Antarctica](https://sites.uci.edu/morlighem/bedmachine-antarctica/) using mass conservation, etc <img src="https://www.pgc.umn.edu/files/2018/08/REMA-hillshade-rendering-800px-768x768.jpg" alt="Hillshade Map of the Reference Elevation Model of Antarctica from Howat et al. 2018" style="margin:0px auto;display:block" width="25%"/> <small>Hillshade Map of the Reference Elevation Model of Antarctica (REMA) from [Howat et al. 2018](https://doi.org/10.7910/DVN/SAIK8B)</small> ---- ### The idea - get the best of both worlds - **Train** neural network on areas with **high resolution** grid data. - Given high resolution surface datasets + prior knowledge of bed, model learns to predict high resolution bed topography - High resolution groundtruth areas provide 'answer' to train the neural network. X(Surface inputs) -- function(X) --> Y(Groundtruth bed) <span class="fragment fade-in"> - **Apply** trained model to **fill in gaps** where there is few/no survey data X(Surface inputs) -- function(X) --> Y(High Resolution Bed) </span> --- ### Inverse models - an ill-posed problem <span class="fragment fade-in"> - Removing data is easy -> **High** to **Low** resolution image - Adding data is difficult -> **Low** to **High** resolution image <img src="https://hoya012.github.io/assets/img/deep_learning_super_resolution/2.PNG" alt="Low to High Resolution is an ill-posed problem" style="margin:0px auto;display:block" width="45%"/> Super-Resolution is one of these hard problems, how do we produce a **realistic** high resolution image from a low resolution image. </span> ---- ### Towards Generative Adversarial Network (GAN) models <img src="https://ieeexplore.ieee.org/mediastore_new/IEEE/content/media/8097368/8099483/8099502/8099502-fig-3-source-large.gif" alt="MSE vs GAN based methods in Ledig et al., 2017 paper" style="margin:0px auto;display:block" width="40%"/> Why? Because GANs can drive the reconstruction towards a more 'natural' look, compared to standard ConvNets that simply reduce the Mean Squared Error (MSE) loss. ---- ### Generative Adversarial Network intuition Two competing neural networks working to improve image's finer details <img src="https://user-images.githubusercontent.com/23487320/162362778-f62158d4-0633-4010-b6cb-aa294146e83e.png" alt="Generative Adversarial Network mechanism from https://www.uv.es/gonmagar/talks/https://docs.google.com/presentation/d/1gMVuW7j6CAAha8Zzkjq8lhq85_9zB0lsOxQ5Vko9cGI/edit?usp=sharing" width="55%"> Generator (artist) learns to produce better image to convince Discriminator, Discriminator (teacher) points out where image is incorrect <small>https://towardsdatascience.com/intuitive-introduction-to-generative-adversarial-networks-gans-230e76f973a9</small> ---- **2016-2017**: Super Resolution Generative Adversarial Network (SRGAN) by [Ledig et al., 2017](https://doi.org/10.1109/CVPR.2017.19) <img src="https://ieeexplore.ieee.org/mediastore_new/IEEE/content/media/8097368/8099483/8099502/8099502-fig-4-source-large.gif" alt="SRGAN architecture in Ledig et al., 2017 paper" style="margin:0px auto;display:block" width="60%"/> Generator-Discriminator GAN models have parallels with Actor-Critic models in Reinforcement Learning. ---- **2018-2019**: Enhanced Super Resolution Generative Adversarial Network (ESRGAN) by [Wang et al., 2019](https://doi.org/10.1007/978-3-030-11021-5_5) <img src="https://media.springernature.com/lw785/springer-static/image/chp%3A10.1007%2F978-3-030-11021-5_5/MediaObjects/478826_1_En_5_Fig3_HTML.png" alt="ESRGAN architecture in Wang et al., 2019 paper" style="margin:0px auto;display:block" width="90%"/> <img src="https://media.springernature.com/original/springer-static/image/chp%3A10.1007%2F978-3-030-11021-5_5/MediaObjects/478826_1_En_5_Fig4_HTML.png" alt="ESRGAN architecture in Wang et al., 2019 paper" style="margin:0px auto;display:block" width="90%"/> ---- ### Extra help - use 'Network Conditioning' to get more info - Since we're not working with 'typical' photographs, we don't need to do *Single Image* Super Resolution. - Additional context (i.e. geographical layers) can be added to produce better Super-Resolution results! ---- Pan-sharpening is a classic (remote-sensing) example. Given a **high resolution** panchromatic band and **low resolution** RGB bands -> produce a **'super resolution'** RGB image. <img src="https://ieeexplore.ieee.org/mediastore_new/IEEE/content/media/8436606/8451009/8451049/0000873-fig-1-source-large.gif" alt="PSGAN results in Liu et al., 2018 paper" style="margin:0px auto;display:block" width="60%"/> <small>Figure showing pan-sharpened results using PSGAN from [Liu et al., 2018](https://doi.org/10.1109/ICIP.2018.8451049)</small> --- ### Bringing it together ---- <section data-visibility="hidden" data-visibility="uncounted">DeepBedMap Model Schematic</section> <!-- <img src="https://yuml.me/diagram/scruffy;dir:LR/class/[BEDMAP2 (1000m){bg:turquoise}]->[Generator model],[REMA Ice Surface Elevation (100m)]->[Generator model],[MEASURES Ice Flow Velocity (450m)]->[Generator model],,[Antarctic Snow Accumulation (1000m)]->[Generator model],[Generator model]->[DeepBedMap DEM (250m){bg:royalblue}],[DeepBedMap DEM (250m)]->[Discriminator model],[Groundtruth Image (250m)]->[Discriminator model],[Discriminator model]->[Real/Fake]" alt="4 input ESRGAN model"/> <small>Input Feature Extraction -> Super Resolution Image Generation -> Output judged by Discriminator</small> --> ---- <section data-visibility="hidden" data-visibility="uncounted">Some equations</section> <!-- Inputs inspired by mass conservation equation: $\frac{dh}{dt} = \frac{dm}{dt} - \nabla \cdot h \textbf{v}$ where change in ice thickness $\frac{dh}{dt}$ equals change in mass balance over time $\frac{dm}{dt}$ (accumulation $c$ - ablation $a$) minus divergence $\nabla \cdot$ in ice thickness $h$ (surface elev $z_s$ - bed elev $z_b$) multiplied by depth-averaged horizontal velocity $\textbf{v}$ (~equal to ice surface velocity). Currently the model includes BEDMAP2 $z_b$, REMA $z_s$, MEASURES Ice Surface Velocity $v$ and Antarctic Snow Accumulation $c$. 'Assumes' steady state for ice thickness change $\frac{dh}{dt}$ and ignoring ablation $a$ term which is difficult to obtain. Also not considering influence of firn compaction or bed-elevation change from tectonic activity. --> ---- ### DeepBedMap Generator Model Architecture <img src="https://tc.copernicus.org/articles/14/3687/2020/tc-14-3687-2020-f01-web.png" alt="ESRGAN architecture - Generator Network" style="margin:0px auto;display:block" width="50%"/> <small>Model adapted from [ESRGAN](https://doi.org/10.1007/978-3-030-11021-5_5) and built using Chainer (Python deep learning library)</small> ---- <section data-visibility="hidden" data-visibility="uncounted">Architecture description</section> <!-- - Input images are on the left, includes prior low resolution bed image, and conditioning inputs. - Core Module are Residual Dense Block layers - Upsampling Module consists of nearest neighbour upsampling followed by deformable convolution layers - Trained using a custom weighted loss function = Content Loss (MSE) + Adversarial Loss + Topographic Loss + Structural Loss (Structural Similarity Index Metric looking at luminance, contrast, and structural info. --> ---- <section data-visibility="hidden" data-visibility="uncounted">Inputs into trained Neural Network Model</section> <!--  --> ---- ### Super Resolution results (4x upsampling) <small>Example over Pine Island Glacier.</small>  --- ### Visualizing the model training <small>On a test area over Thwaites Glacier.</small> <table align="center"> <tr> <td> <img style="width: 400px; height: 450px" src="https://user-images.githubusercontent.com/23487320/96398523-9142d480-1228-11eb-8ee3-2b574c542b57.png"/> <a>Epoch 1</a> </td> <td> <img style="width: 400px; height: 450px" src="https://user-images.githubusercontent.com/23487320/96398734-0c0bef80-1229-11eb-8720-9570b444fbcf.gif"/> <a>Training</a> </td> <td> <img style="width: 400px; height: 450px" src="https://user-images.githubusercontent.com/23487320/96398791-33fb5300-1229-11eb-8ae0-a006203533de.png"/> <a>Epoch 100</a> </td> </tr> </table> ---- ### Along transect elevation and roughness <img src="https://tc.copernicus.org/articles/14/3687/2020/tc-14-3687-2020-f06-web.png" alt="Plot of elevation and roughness values along a transect" style="margin:0px auto;display:block" width="42%"/> <small>DeepBedMap_DEM (purple) features more fine-scale (<10km) bumps and troughs, and higher roughness values (mean standard deviation of about 40m), similar to the ground truth (orange)</small> ---- ### Hyperparameter Tuning <small>To optimize the ESRGAN model's performance, a **Bayesian** approach (Tree-Structured Parzen Estimator) was used to narrow down our **hyperparameter search** space.</small> <table> <tr> <td> <img style="width: 350px; height: 250px" src="https://66.media.tumblr.com/214e16d057ee4a1ea3e0cfd8f6ca204d/tumblr_inline_pj9ipgmdkg1toi3ym_500.gif"/> <center><small>Grid Search</small></center> </td> <td> <img style="width: 350px; height: 250px" src="https://66.media.tumblr.com/265cfe10bcda2018ce90ed2d060fabd7/tumblr_inline_pj9iphWyRH1toi3ym_500.gif"/> <center><small>Random Search</small></center> </td> <td> <img style="width: 350px; height: 250px" src="https://66.media.tumblr.com/943075996f8454b041238d31ec6671fb/tumblr_inline_pj9iphFzyu1toi3ym_500.gif"/> <center><small>Bayesian Optimization</small></center> </td> </tr> </table> <small>[HyperBand](http://arxiv.org/abs/1603.06560) used to prune unpromising trials.</small> <small>Main 'hyperparameters' tuned (in rough order of priority) were:</small> <small>Learning rate (**1.7e-4**, 2e-4 to 1e-4); Residual scaling factor (**0.2**, 0.1 to 0.5); Training epochs (**~140**, 90 to 150); Number of Residual-in-Residual Dense Blocks (**12**, 8 to 14); Mini-batch size (**128**, 64 or 128)</small> ---- <img src="https://github.com/weiji14/deepbedmap/releases/download/v1.1.0/fig0_deepbedmap_dem.png" alt="DeepBedMap DEM over entire Antarctic continent, so beautiful~" style="margin:0px auto;display:block" width="60%"/> <p></p> <center><small>1-2 days to train (and fine-tune), less than 1 minute to run for the whole continent (on a Tesla V100 GPU).</small></center> --- ### :rocket: Moving forward :rocket: - Better data => Better model - Need more **high resolution** swath radar surveys (<= 250 m flight spacing) - waiting for BEDMAP3!! - Fill remote sensing **data gaps** (e.g. due to clouds) - Update model with domain specific components - **Modular** design means different pieces can be 'upgraded' as new architectures come online - **Combine** super-resolution with glaciology mass conservation techniques (DeepBedMap + BedMachine = DeepBedMapV2?) <small>P.S. Slides are at https://hackmd.io/@weiji14/2022ML4Polar and DeepBedMap paper is at https://doi.org/10.5194/tc-14-3687-2020. Or check out the repo at https://github.com/weiji14/deepbedmap.</small> ---- ### References (1) - Fretwell, P., Pritchard, H. D., Vaughan, D. G., Bamber, J. L., Barrand, N. E., Bell, R., … Zirizzotti, A. (2013). Bedmap2: improved ice bed, surface and thickness datasets for Antarctica. The Cryosphere, 7(1), 375–393. https://doi.org/10.5194/tc-7-375-2013 - Gardner, A. S., Moholdt, G., Scambos, T., Fahnstock, M., Ligtenberg, S., van den Broeke, M., & Nilsson, J. (2018). Increased West Antarctic and unchanged East Antarctic ice discharge over the last 7 years. The Cryosphere, 12(2), 521–547. https://doi.org/10.5194/tc-12-521-2018 - Howat, Ian, Morin, Paul, Porter, Claire, & Noh, Myong-Jong. (2018). The Reference Elevation Model of Antarctica [Data set]. Harvard Dataverse. https://doi.org/10.7910/DVN/SAIK8B ---- ### References (2) - Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., & Shi, W. (2017). Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 105–114. https://doi.org/10.1109/CVPR.2017.19 - Leong, W. J., & Horgan, H. J. (2020). DeepBedMap: A deep neural network for resolving the bed topography of Antarctica. The Cryosphere, 14(11), 3687–3705. https://doi.org/10.5194/tc-14-3687-2020 ---- ### References (3) - Morlighem, M., Rignot, E., Binder, T., Blankenship, D., Drews, R., Eagles, G., Eisen, O., Ferraccioli, F., Forsberg, R., Fretwell, P., Goel, V., Greenbaum, J. S., Gudmundsson, G. H., Guo, J., Helm, V., Hofstede, C., Howat, I., Humbert, A., Jokat, W., … Young, D. A. (2019). Deep glacial troughs and stabilizing ridges unveiled beneath the margins of the Antarctic ice sheet. Nature Geoscience, 13(2), 132–137. https://doi.org/10.1038/s41561-019-0510-8 - Wang, X., Yu, K., Wu, S., Gu, J., Liu, Y., Dong, C., … Tang, X. (2018). ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks. ArXiv:1809.00219 [Cs]. Retrieved from http://arxiv.org/abs/1809.00219