---
title: "DeepBedMap: A Super-Resolution Generative Adversarial Network for resolving the subglacial topography of Antarctica"
description: Machine Learning for Polar Regions Workshop presentation
tags: Talk
type: slide
slideOptions:
theme: simple
width: 75%
---

<!--
Docs for making Markdown slide deck on HackMD using Revealjs
https://hackmd.io/s/how-to-create-slide-deck
https://revealjs.com
-->

### :airplane_departure:+:satellite:+:computer: = :snow_capped_mountain: :flag-aq:

#### **DeepBedMap: A Super-Resolution GAN for resolving the subglacial topography of Antarctica**

<small> Machine Learning for Polar Regions Workshop presentation at
Lamont-Doherty Earth Observatory (virtual), Friday 17 Jun 2022, 15:15-15:30 (UTC) </small>

_by **[Wei Ji Leong](https://github.com/weiji14)** & [Huw Horgan](https://orcid.org/0000-0002-4836-0078)_

<!-- Put the link to this slide here so people can follow -->
<small> P.S. Slides are at https://hackmd.io/@weiji14/2022ML4Polar</small>

---

### Why do we need high resolution images?

Going from a 1km resolution BEDMAP2 to a higher (250m) spatial resolution **bed topography** would enable us to:

<span class="fragment fade-in">

- [x] Capture more glaciological processes in **ice sheet models**
- Ice flows slower over rough beds compared to smooth beds

</span>

<span class="fragment fade-in">

- [x] Better understand Antarctica's **subglacial hydrology**
- Water under the ice can lower friction and lead to faster flow

</span>

<span class="fragment fade-in">

Cumulatively, this will lead to more accurate **sea level rise** :ocean: projections!

</span>

---

### How to get a better BEDMAP?

----

### Direct approach

- **Ice-penetrating radar** surveys, accurate but **limited spatial coverage**
- E.g. from British Antarctic Survey, Operation Icebridge field missions, etc

<img src="https://user-images.githubusercontent.com/23487320/49407666-392e9080-f7be-11e8-9788-061dc5040796.png" alt="Map of radio-echo-sounding datasets around Antarctica used in Gardner et al., 2018 paper" style="margin:0px auto;display:block" width="35%"/>

<small>Figure showing Radio-echo-sounding datasets around Antarctica from [Gardner et al. 2018](https://doi.org/10.5194/tc-12-521-2018)</small>

----

### Indirect approach

- **Inverse** model on satellite captured **ice surface** data, less accurate but **widely applicable**
- E.g. Our [**DeepBedMap**](https://doi.org/10.5194/tc-2020-74) model, [BedMachine Antarctica](https://sites.uci.edu/morlighem/bedmachine-antarctica/) using mass conservation, etc

<img src="https://www.pgc.umn.edu/files/2018/08/REMA-hillshade-rendering-800px-768x768.jpg" alt="Hillshade Map of the Reference Elevation Model of Antarctica from Howat et al. 2018" style="margin:0px auto;display:block" width="25%"/>

<small>Hillshade Map of the Reference Elevation Model of Antarctica (REMA) from [Howat et al. 2018](https://doi.org/10.7910/DVN/SAIK8B)</small>

----

### The idea - get the best of both worlds

- **Train** neural network on areas with **high resolution** grid data.
- Given high resolution surface datasets + prior knowledge of bed, model learns to predict high resolution bed topography
- High resolution groundtruth areas provide 'answer' to train the neural network.

X(Surface inputs) -- function(X) --> Y(Groundtruth bed)

<span class="fragment fade-in">

- **Apply** trained model to **fill in gaps** where there is few/no survey data

X(Surface inputs) -- function(X) --> Y(High Resolution Bed)

</span>

---

### Inverse models - an ill-posed problem

<span class="fragment fade-in">

- Removing data is easy -> **High** to **Low** resolution image
- Adding data is difficult -> **Low** to **High** resolution image

<img src="https://hoya012.github.io/assets/img/deep_learning_super_resolution/2.PNG" alt="Low to High Resolution is an ill-posed problem" style="margin:0px auto;display:block" width="45%"/>

Super-Resolution is one of these hard problems, how do we produce a **realistic** high resolution image from a low resolution image.

</span>

----

### Towards Generative Adversarial Network (GAN) models

<img src="https://ieeexplore.ieee.org/mediastore_new/IEEE/content/media/8097368/8099483/8099502/8099502-fig-3-source-large.gif" alt="MSE vs GAN based methods in Ledig et al., 2017 paper" style="margin:0px auto;display:block" width="40%"/>

Why? Because GANs can drive the reconstruction towards a more 'natural' look, compared to standard ConvNets that simply reduce the Mean Squared Error (MSE) loss.

----

### Generative Adversarial Network intuition

Two competing neural networks working to improve image's finer details

<img src="https://user-images.githubusercontent.com/23487320/162362778-f62158d4-0633-4010-b6cb-aa294146e83e.png" alt="Generative Adversarial Network mechanism from https://www.uv.es/gonmagar/talks/https://docs.google.com/presentation/d/1gMVuW7j6CAAha8Zzkjq8lhq85_9zB0lsOxQ5Vko9cGI/edit?usp=sharing" width="55%">

Generator (artist) learns to produce better image to convince Discriminator, Discriminator (teacher) points out where image is incorrect

<small>https://towardsdatascience.com/intuitive-introduction-to-generative-adversarial-networks-gans-230e76f973a9</small>

----

**2016-2017**: Super Resolution Generative Adversarial Network (SRGAN) by [Ledig et al., 2017](https://doi.org/10.1109/CVPR.2017.19)

<img src="https://ieeexplore.ieee.org/mediastore_new/IEEE/content/media/8097368/8099483/8099502/8099502-fig-4-source-large.gif" alt="SRGAN architecture in Ledig et al., 2017 paper" style="margin:0px auto;display:block" width="60%"/>

Generator-Discriminator GAN models have parallels with Actor-Critic models in Reinforcement Learning.

----

**2018-2019**: Enhanced Super Resolution Generative Adversarial Network (ESRGAN) by [Wang et al., 2019](https://doi.org/10.1007/978-3-030-11021-5_5)

<img src="https://media.springernature.com/lw785/springer-static/image/chp%3A10.1007%2F978-3-030-11021-5_5/MediaObjects/478826_1_En_5_Fig3_HTML.png" alt="ESRGAN architecture in Wang et al., 2019 paper" style="margin:0px auto;display:block" width="90%"/>

<img src="https://media.springernature.com/original/springer-static/image/chp%3A10.1007%2F978-3-030-11021-5_5/MediaObjects/478826_1_En_5_Fig4_HTML.png" alt="ESRGAN architecture in Wang et al., 2019 paper" style="margin:0px auto;display:block" width="90%"/>

----

### Extra help - use 'Network Conditioning' to get more info

- Since we're not working with 'typical' photographs, we don't need to do *Single Image* Super Resolution.
- Additional context (i.e. geographical layers) can be added to produce better Super-Resolution results!

----

Pan-sharpening is a classic (remote-sensing) example. Given a **high resolution** panchromatic band and **low resolution** RGB bands -> produce a **'super resolution'** RGB image.

<img src="https://ieeexplore.ieee.org/mediastore_new/IEEE/content/media/8436606/8451009/8451049/0000873-fig-1-source-large.gif" alt="PSGAN results in Liu et al., 2018 paper" style="margin:0px auto;display:block" width="60%"/>

<small>Figure showing pan-sharpened results using PSGAN from [Liu et al., 2018](https://doi.org/10.1109/ICIP.2018.8451049)</small>

---

### Bringing it together

----

<section data-visibility="hidden" data-visibility="uncounted">DeepBedMap Model Schematic</section>

<!--

<img src="https://yuml.me/diagram/scruffy;dir:LR/class/[BEDMAP2 (1000m){bg:turquoise}]->[Generator model],[REMA Ice Surface Elevation (100m)]->[Generator model],[MEASURES Ice Flow Velocity (450m)]->[Generator model],,[Antarctic Snow Accumulation (1000m)]->[Generator model],[Generator model]->[DeepBedMap DEM (250m){bg:royalblue}],[DeepBedMap DEM (250m)]->[Discriminator model],[Groundtruth Image (250m)]->[Discriminator model],[Discriminator model]->[Real/Fake]" alt="4 input ESRGAN model"/>

<small>Input Feature Extraction -> Super Resolution Image Generation -> Output judged by Discriminator</small>
-->
----

<section data-visibility="hidden" data-visibility="uncounted">Some equations</section>

<!--
Inputs inspired by mass conservation equation: $\frac{dh}{dt} = \frac{dm}{dt} - \nabla \cdot h \textbf{v}$

where change in ice thickness $\frac{dh}{dt}$ equals
change in mass balance over time $\frac{dm}{dt}$ (accumulation $c$ - ablation $a$) minus
divergence $\nabla \cdot$ in ice thickness $h$ (surface elev $z_s$ - bed elev $z_b$) multiplied by
depth-averaged horizontal velocity $\textbf{v}$ (~equal to ice surface velocity).

Currently the model includes BEDMAP2 $z_b$, REMA $z_s$, MEASURES Ice Surface Velocity $v$ and Antarctic Snow Accumulation $c$.
'Assumes' steady state for ice thickness change $\frac{dh}{dt}$ and ignoring ablation $a$ term which is difficult to obtain.
Also not considering influence of firn compaction or bed-elevation change from tectonic activity.

-->
----

### DeepBedMap Generator Model Architecture

<img src="https://tc.copernicus.org/articles/14/3687/2020/tc-14-3687-2020-f01-web.png" alt="ESRGAN architecture - Generator Network" style="margin:0px auto;display:block" width="50%"/>

<small>Model adapted from [ESRGAN](https://doi.org/10.1007/978-3-030-11021-5_5) and built using Chainer (Python deep learning library)</small>

----

<section data-visibility="hidden" data-visibility="uncounted">Architecture description</section>

<!--
- Input images are on the left, includes prior low resolution bed image, and conditioning inputs.
- Core Module are Residual Dense Block layers
- Upsampling Module consists of nearest neighbour upsampling followed by deformable convolution layers

- Trained using a custom weighted loss function = Content Loss (MSE) + Adversarial Loss + Topographic Loss + Structural Loss (Structural Similarity Index Metric looking at luminance, contrast, and structural info.

-->
----

<section data-visibility="hidden" data-visibility="uncounted">Inputs into trained Neural Network Model</section>

<!-- 
-->
----

### Super Resolution results (4x upsampling)

<small>Example over Pine Island Glacier.</small>


---

### Visualizing the model training

<small>On a test area over Thwaites Glacier.</small>

<table align="center">
<tr>
<td>
<img style="width: 400px; height: 450px" src="https://user-images.githubusercontent.com/23487320/96398523-9142d480-1228-11eb-8ee3-2b574c542b57.png"/>
<a>Epoch 1</a>
</td>
<td>
<img style="width: 400px; height: 450px" src="https://user-images.githubusercontent.com/23487320/96398734-0c0bef80-1229-11eb-8720-9570b444fbcf.gif"/>
<a>Training</a>
</td>
<td>
<img style="width: 400px; height: 450px" src="https://user-images.githubusercontent.com/23487320/96398791-33fb5300-1229-11eb-8ae0-a006203533de.png"/>
<a>Epoch 100</a>
</td>
</tr>
</table>

----

### Along transect elevation and roughness

<img src="https://tc.copernicus.org/articles/14/3687/2020/tc-14-3687-2020-f06-web.png" alt="Plot of elevation and roughness values along a transect" style="margin:0px auto;display:block" width="42%"/>

<small>DeepBedMap_DEM (purple) features more fine-scale (<10km) bumps and troughs, and higher roughness values (mean standard deviation of about 40m), similar to the ground truth (orange)</small>

----

### Hyperparameter Tuning

<small>To optimize the ESRGAN model's performance, a **Bayesian** approach (Tree-Structured Parzen Estimator) was used to narrow down our **hyperparameter search** space.</small>

<table>
<tr>
<td>
<img style="width: 350px; height: 250px" src="https://66.media.tumblr.com/214e16d057ee4a1ea3e0cfd8f6ca204d/tumblr_inline_pj9ipgmdkg1toi3ym_500.gif"/>
<center><small>Grid Search</small></center>
</td>
<td>
<img style="width: 350px; height: 250px" src="https://66.media.tumblr.com/265cfe10bcda2018ce90ed2d060fabd7/tumblr_inline_pj9iphWyRH1toi3ym_500.gif"/>
<center><small>Random Search</small></center>
</td>
<td>
<img style="width: 350px; height: 250px" src="https://66.media.tumblr.com/943075996f8454b041238d31ec6671fb/tumblr_inline_pj9iphFzyu1toi3ym_500.gif"/>
<center><small>Bayesian Optimization</small></center>
</td>
</tr>
</table>

<small>[HyperBand](http://arxiv.org/abs/1603.06560) used to prune unpromising trials.</small>
<small>Main 'hyperparameters' tuned (in rough order of priority) were:</small>
<small>Learning rate (**1.7e-4**, 2e-4 to 1e-4); Residual scaling factor (**0.2**, 0.1 to 0.5); Training epochs (**~140**, 90 to 150); Number of Residual-in-Residual Dense Blocks (**12**, 8 to 14); Mini-batch size (**128**, 64 or 128)</small>

----

<img src="https://github.com/weiji14/deepbedmap/releases/download/v1.1.0/fig0_deepbedmap_dem.png" alt="DeepBedMap DEM over entire Antarctic continent, so beautiful~" style="margin:0px auto;display:block" width="60%"/>

<p></p>
<center><small>1-2 days to train (and fine-tune), less than 1 minute to run for the whole continent (on a Tesla V100 GPU).</small></center>

---

### :rocket: Moving forward :rocket:

- Better data => Better model
- Need more **high resolution** swath radar surveys (<= 250 m flight spacing) - waiting for BEDMAP3!!
- Fill remote sensing **data gaps** (e.g. due to clouds)
- Update model with domain specific components
- **Modular** design means different pieces can be 'upgraded' as new architectures come online
- **Combine** super-resolution with glaciology mass conservation techniques (DeepBedMap + BedMachine = DeepBedMapV2?)

<small>P.S. Slides are at https://hackmd.io/@weiji14/2022ML4Polar and DeepBedMap paper is at https://doi.org/10.5194/tc-14-3687-2020. Or check out the repo at https://github.com/weiji14/deepbedmap.</small>

----

### References (1)

- Fretwell, P., Pritchard, H. D., Vaughan, D. G., Bamber, J. L., Barrand, N. E., Bell, R., … Zirizzotti, A. (2013). Bedmap2: improved ice bed, surface and thickness datasets for Antarctica. The Cryosphere, 7(1), 375–393. https://doi.org/10.5194/tc-7-375-2013

- Gardner, A. S., Moholdt, G., Scambos, T., Fahnstock, M., Ligtenberg, S., van den Broeke, M., & Nilsson, J. (2018). Increased West Antarctic and unchanged East Antarctic ice discharge over the last 7 years. The Cryosphere, 12(2), 521–547. https://doi.org/10.5194/tc-12-521-2018

- Howat, Ian, Morin, Paul, Porter, Claire, & Noh, Myong-Jong. (2018). The Reference Elevation Model of Antarctica [Data set]. Harvard Dataverse. https://doi.org/10.7910/DVN/SAIK8B

----

### References (2)

- Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., & Shi, W. (2017). Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 105–114. https://doi.org/10.1109/CVPR.2017.19

- Leong, W. J., & Horgan, H. J. (2020). DeepBedMap: A deep neural network for resolving the bed topography of Antarctica. The Cryosphere, 14(11), 3687–3705. https://doi.org/10.5194/tc-14-3687-2020

----

### References (3)

- Morlighem, M., Rignot, E., Binder, T., Blankenship, D., Drews, R., Eagles, G., Eisen, O., Ferraccioli, F., Forsberg, R., Fretwell, P., Goel, V., Greenbaum, J. S., Gudmundsson, G. H., Guo, J., Helm, V., Hofstede, C., Howat, I., Humbert, A., Jokat, W., … Young, D. A. (2019). Deep glacial troughs and stabilizing ridges unveiled beneath the margins of the Antarctic ice sheet. Nature Geoscience, 13(2), 132–137. https://doi.org/10.1038/s41561-019-0510-8

- Wang, X., Yu, K., Wu, S., Gu, J., Liu, Y., Dong, C., … Tang, X. (2018). ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks. ArXiv:1809.00219 [Cs]. Retrieved from http://arxiv.org/abs/1809.00219