# Hydroviet Project

**Principal Investigator** Phong Dinh Vo ([e-mail])

*Last Updated* 8th November 2021

## Topics

### 1. Visual Object Retrieval from EO Data (VOR)

Retrieve visual structures, including man-made and natural, at the country-wide level; for example find hydropower plants, constructions in forest, households in rural or mountainous areas, bridges, fords, streams, concrete high buildings, high grounds, landslide sites.

A good object retrieval algorithm allows searching at large-scale efficiently, updating object database directly from imagery, doing change detection on the same site over time. Object localization and classification is an indispensable component in humanitarian relief efforts. Households in high risk areas could be put in evacuation or rescue plans. Object localization from imagery also by-pass the need of having access to administrative databases which are often difficult to obtain.

#### Research approach

1. To get the high resolution imagery to work with, Bing Images could be a good source for non-commercial purposes.
2. Self-supervised learning on a large-scale supervised satellite image database and use the model as a feature extractor
3. Integrate the model with FAISS and Milvus to provide an efficient image search service with complete infrastructure backends and front end graphical web interface.
4. Develop an RESTfull API or Python package to provide image search online

#### Expected contributions

1. Develop a good open-source Python data pipeline that collects, parses, processs and indexes satellite imagery on demand, geolocation-based.
2. Train a good neural network for image representation based on satellite image data in Vietnam and similar developing countries
3. Build an image search engine on top of existing techs, up and run.

#### What you will learn

1. Good programming skills on Python, Pytorch or Tensorflow
2. Big data image processing and feature extraction, SQL databases
3. Deep learning practices for image domain
4. Image search / retrieval technologies

#### Good fit for you?

1. If you love software engineering, backend & frontend, docker, databases
2. If you want to use deep learning in interesting applications
3. If you love working with big data and distributed computing

#### References

1. [Predicting Ground-Level Scene Layout from Aerial Imagery](https://arxiv.org/abs/1612.02709)
2. [Terra Pattern](https://github.com/CreativeInquiry/terrapattern)
3. [Deep OpenStreetMap](https://github.com/trailbehind/DeepOSM)
4. [FAISS](https://github.com/facebookresearch/faiss)
5. [Milvus - An open source vector similarity Search Engine](https://milvus.io/)
6. [Mapping the world to help aid workers, with weakly, semi-supervised learning](https://ai.facebook.com/blog/mapping-the-world-to-help-aid-workers-with-weakly-semi-supervised-learning/)

### 2. Visual Object Categorization from Satellite Images (VOC)

Categorizing man-made structures and natural objects are one step closer to accurate loss assessment and humanitarian aids in catastrophic events. This topic focuses on categorizing building types based on bird-view images and angled views if available. Inferring building heights are useful in flood events.

#### Research approach

1. Based on Open Street Map and other sources to get object annotation and labels.
2. Given object boundary and location, classifying them into known categories using convolutional neural net models, focusing on flood-prone areas.
3. Attempt to infer structure heights

#### Expected contributions

1. Visual object classification model based on deep networks on hundreds of categories with good accuracy
2. Building segmentation from satellite images with good accuracy
3. A building height predictor which is useful in flood simulation

#### What you will learn

1. Master in visual object classification problems
2. Skills to define and solve open-set classification problems, using open-source GIS data
3. Good skills in deep learning for object classification and segmentation
4. Good skills on Python, Pytorch / Tensorflow

#### Good fit for you?

1. If you love visual cognition problem, vision intelligence too
2. If you want to dive deep into deep network models for images
3. If you can afford to run heavy models on GPUs for days

#### References

1. [Topological Map Extraction From Overhead Images](https://arxiv.org/abs/1812.01497)
2. [Detecting Roads from Satellite Imagery in the Developing World](https://openaccess.thecvf.com/content_CVPRW_2019/papers/cv4gc/Nachmany_Detecting_Roads_from_Satellite_Imagery_in_the_Developing_World_CVPRW_2019_paper.pdf)
3. [Learning to Detect Roads in High-Resolution Aerial Images](https://www.cs.toronto.edu/~hinton/absps/road_detection.pdf)
4. [SpaceNet.AI](https://spacenet.ai)
5. [Combining satellite imagery and machine learning to predict poverty](https://forum.stanford.edu/events/posterslides/CombiningSatelliteImageryandMachineLearningtoPredictPoverty.pdf)
6. [Deep OpenStreetMap](https://github.com/trailbehind/DeepOSM)
7. [Building Detection from Satellite Imagery using Ensemble of Size-specific Detectors](https://openaccess.thecvf.com/content_cvpr_2018_workshops/papers/w4/Hamaguchi_Building_Detection_From_CVPR_2018_paper.pdf)
8. [Radiant Earth Foundation](https://www.radiant.earth/)

### 3. Inland Water Monitoring and Prediction (WMP)

Monitor water levels of inland water bodies including reservoirs and hydropower dams and water ways. Based on Digital Elevation Model (DEM) combined with SAR Sentinel-1 imagery, find insights from annual and seasonal hydrological cycles in the region, and combine with weather data of wind, rainfall, water runoff, temperature, etc. to improve monitoring accuracy, and early prediction.

To improve water segmentation quality, reaching to higher spatial resolution along the shores is desirable, given that Sentinel images are already at 10m resolution. To complete the picture, DEM products based on SRTM data could be extrapolated to higher-resolution, from the original 30m (1 arcsecond) up to 10m, or even more detailed. To this end, SRTM DEM could be fused with other DEM sources including DEM with better resolution, LiDAR DEM. For there is no guarantees of availability of different DEM products at the same geographical location, super-resolution transfer learning is a good idea to experiment with; in other words, super-resolution DEM is trained in somewhere else and applied into Vietnam.

#### Research approach

1. Continue to develop water monitoring algorithm from the last year, which uses Synthetic Aperture Radar images of Copernicus' Sentinel-1 satellite and Digital Elevation Map of NASA SRTM program; Improve algorithm robustness on detected water boundaries.
2. Extrapolate DEM at near-shore underwater areas to monitor water level in drought years.
3. DEM Super-resolution: to fuse 10m DEM images and under 1m resolution of RGB Earth Observation to achieve higher resolution for DEM images, henceforth improving water level segmentation accuracy. LIDAR images could be used as hi-res DEM groundtruth, then pretrained models could be transfered to new geographical locations.
4. One-timestep ahead prediction based on historical water level data and weather data including rainfall, evaporation, runoff, wind, temperature. Looking for a probabilistic regression model capable of modeling these observations.

#### Expected contributions

1. Water monitoring algorithm works seamlessly on SAR Sentinel-1 imagery
2. Water level forecasting algorithm with multi-modal inputs including historical water levels, precipitation (or rainfall), runoff, evaporation, temperature, wind.
3. Super-resolution DEM transfer learning: coastal / shore areas of reservoirs and high risk flood areas.

#### What you will learn

1. Good skills on remote sensing research, image processing
2. Deep learning with transfer learning problem
3. Image super-resolution problem with deep learning approach
4. Timeseries forecasting and knowledge in weather & climate
5. Good skills on Python, Pytorch / TF, pandas, sklearn, skimage

#### Good fit for you?

1. If you love planetary science, earth observation engineering, if you are obssessed with spacecrafts and physics
2. If you have background in signal processing
3. If you want to see more applications of deep learning, beyond recognizing dogs and cats
4. If you want to give a direct impact on how to save people's lives from floods

#### References

1. [D-SRGAN: DEM Super-Resolution with Generative Adversarial Networks](https://eartharxiv.org/repository/view/305/)
2. [LoGSRN: Deep Super Resolution Network for Digital Elevation Model](https://sa.catapult.org.uk/digital-library/logsrn-deep-super-resolution-network-for-digital-elevation-model/)
3. [SentinelHub - Water Resources Monitoring](https://www.sentinel-hub.com/explore/industries-and-showcases/water-resources-monitoring/)
4. [CoastalDEM: A global coastal digital elevation model improved from SRTM using a neural network](https://www.sciencedirect.com/science/article/abs/pii/S0034425717306016)
5. [ECMWF Climate Data Store](https://cds.climate.copernicus.eu/cdsapp#!/home)

### 4. Change Detection for Catastrophic Loss Assessment (CDL)

Humanitatian aid requires a good loss assessment so that right helps come to right places, right people. In catastrophic events like floods, aerial images including satellite observations are vital to localize damages, and to assess the severity of losses. For heavy rainfalls and flooding are often associated with bad weather, satellite visibility is severely affected because of heavy clouds. Synthetic Aperture Radar images stand out as the unique way "seeing through" the clouds, henceforth effectively detect flooded regions.

Change detection plays vital roles in identifying and localizing damaged areas or places that need supports and rescues. Images captured at the times of before and after the catastrophic event are compared one from another, to find exact locations of affection.

#### Research approach

1. Build up a database of flood events nationalwide from various sources, including building an NLP-based toolkit to retrieve flood events from media.
2. Detect changes due to flood: compare before and after SAR images, combining with optical hi-res satellite image to localize possible damaged households and buildings. Based on change detection networks and siamese networks to propose a new design suitable for SAR and EO images.

#### Expected contributions

1. Python-based NLP crawling & extraction tool for catastrophic news in Vietnamese language
2. Deep learning model capable of detecting flood areas from SAR images
3. A framework (algorithm or a set of algorithms) to estimate damages in flooded areas, including buildings, fields, and roads.

#### What you will learn

1. Expertise knowledge in remote sensing and radar imagery
2. Deep learning with methods on change detection, applicable to wide ranges of areas including urban planning, forest management, intelligence, defense industry.
3. Skill with natural language processing and tools
4. GIS skills and knowledges

#### References

1. [PhoBERT](PhoBERT)
2. [Natural Language Toolkit](https://www.nltk.org/)
2. [The International Disasters Database](https://www.emdat.be/)
3. [Rapid flood and damage mapping using synthetic aperture radar in response to Typhoon Hagibis, Japan](https://www.nature.com/articles/s41597-020-0443-5)
4. [Unsupervised Change Detection in Satellite Images Using Convolutional Neural Networks](https://github.com/annabosman/UNet-based-Unsupervised-Change-Detection)
5. [Multi-temporal synthetic aperture radar flood mapping using change detection](https://www.researchgate.net/publication/315980301_Multi-Temporal_SAR_Flood_Mapping_using_Change_Detection)
6. [A Deep Convolutional Coupling Network for Change Detection Based on Heterogeneous Optical and Radar Images](https://ieeexplore.ieee.org/abstract/document/7795259)
7. [Flood Detection in Gaofen-3 SAR Images via Fully Convolutional Networks](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6165191/)
8. [SAR Flood Mapping using Google Earth Engine](https://youtu.be/4Y2giuRPCuc)
9. [SAR for Flood Mapping](https://www.youtube.com/watch?v=QKrG5jYZe10)

### 5. Causal Reasoning on Explaining Catastrophic Events: Landslides and Flooding (CRL)

Landslides are deadly events happening in mountainous areas, killing many thousands of people each year worldwide. Landslides are very difficult to predict ahead due to complex relationship between geographical developments, weather events, precipitation, in which many of these factors could not be observed by satellites. In this study we aim to apply causal machine learning models into explaining historical landslide events, harnessing as many observation variables as possible, including soil composition, terrain, elevation, runoffs, precipitation, vegetation coverage, agriculture and deforestation activities.

#### Research approach

1. Collect, use, and study global landslide databases
2. Learn about causality and causal inference for machine learning
3. Collect context features of landslide events from multiple sources and imagery including EO data, SAR, weather maps, soil maps, deforestation monitoring output, agriculture maps; apply to a causal reasoning model and try to explain major reasons that lead to landslides.
4. Use CI to explain landslides and floods

#### Expected contributions

1. Causality-based method to explain landslides
2. Causality-based method to explan floods
3. Use results from CDL and WML to improve explanations

#### What will you get after this?

1. Mastering causal inference, a new and growing area of machine learning
2. GIS skills and knowledges in disasters prevention
3. Timeseries prediction and data processing skills

#### Good fit for you?

1. If you love working on theoretical problems, or simply love maths
2. If you do not want too much data wrangling and processing
3. If you want to catch the future of AI in the next 5-10 years
4. If you really want to be able to why, when, and how landslides and floods happened

#### References

1. [Judea Pearl - The Book of Why](http://bayes.cs.ucla.edu/WHY/)
2. [Introduction to Causal Inference](https://www.bradyneal.com/causal-inference-course)
3. [Algorithms for Causal Reasoning in Probability Trees](https://arxiv.org/abs/2010.12237)
4. [Landslides @ NASA](https://gpm.nasa.gov/landslides/index.html)
5. [The International Disasters Database](https://www.emdat.be/)
6. [Global Landslide Catalog](https://data.nasa.gov/Earth-Science/Global-Landslide-Catalog/h9d8-neg4)
7. [ECMWF Climate Data Store](https://cds.climate.copernicus.eu/cdsapp#!/home)
8. [Hệ lụy đáng lo từ thủy điện nhỏ](https://tuoitre.vn/he-luy-dang-lo-tu-thuy-dien-nho-20201110082005383.htm)
9. [Landslide identification using machine learning](https://www.sciencedirect.com/science/article/pii/S1674987120300542#undfig1)
10. [Global Soil Grid](https://soilgrids.org/)

## Training

### EO Basic

1.