# UAVSAR Time Series Analysis - MintPy Processing :timer_clock: :airplane:
* :notebook: This binnacle gives us the steps to follow to perform the time series analysis from a specific UAVSAR interferograms stack in the open source software MintPy.
## Prepare working directory :construction_worker:
> Working directory
```javascript=python
/local3/nacho/UAVSAR/Salton/26515/S02_proc
```
Create :new: directory
```javascript=python
!mkdir mintpy
```
```javascript=python
cd mintpy
```
```javascript=python
!pwd
```
```javascript=python
/local3/nacho/UAVSAR/Salton/26515/S02_proc/mintpy
```
* :notebook: If you don't know the total of processors you have in your computer, type:
```javascript=python
!nproc
```
* :notebook: So far, we are ready to run the `smallbaselineApp.py`, as long as, you are not going to perform tropospheric corrections by using `GACOS` (*Generic Atmospheric Correction Online Service for InSAR*) or `ERA5` (*ECMWF Re-Analysis*), since a specific directories are needed, respectivectly.
* **IMPORTANT!** First check the configuration file (`file.cfg`) to comfirm you have the requirements you need on the processing.
## Example directory structure (ISCE/stripmapStack)
```javascript=python
$DATA_DIR/KirishimaAlosAT424F620_630
├── baselines
│ ├── 20060624_20060924.txt
│ ├── 20060624_20061225.txt
│ └── ...
├── geom_reference
│ ├── hgt.rdr
│ ├── hgt.rdr.full.vrt
│ ├── hgt.rdr.full.xml
│ ├── hgt.rdr.vrt
│ ├── hgt.rdr.xml
│ ├── incLocal.rdr
│ ├── incLocal.rdr.full.vrt
│ ├── incLocal.rdr.full.xml
│ ├── incLocal.rdr.vrt
│ ├── incLocal.rdr.xml
│ ├── lat.rdr
│ ├── lat.rdr.full.vrt
│ ├── lat.rdr.full.xml
│ ├── lat.rdr.vrt
│ ├── lat.rdr.xml
│ ├── lon.rdr
│ ├── lon.rdr.full.vrt
│ ├── lon.rdr.full.xml
│ ├── lon.rdr.vrt
│ ├── lon.rdr.xml
│ ├── los.rdr
│ ├── los.rdr.full.vrt
│ ├── los.rdr.full.xml
│ ├── los.rdr.vrt
│ ├── los.rdr.xml
│ ├── shadowMask.rdr
│ ├── shadowMask.rdr.full.vrt
│ ├── shadowMask.rdr.full.xml
│ ├── shadowMask.rdr.vrt
│ └── shadowMask.rdr.xml
├── merged
│ ├── baselines
│ │ ├── 20060624
│ │ │ ├── 20060624
│ │ │ ├── 20060624.full.vrt
│ │ │ ├── 20060624.vrt
│ │ │ └── 20060624.xml
│ │ ├── 20060924
│ │ │ ├── 20060924
│ │ │ ├── 20060924.full.vrt
│ │ │ ├── 20060924.vrt
│ │ │ └── 20060924.xml
│ │ └── ...
│ └── SLC
│ ├── 20060624
│ │ ├── ...
│ │ └── referenceShelve
│ │ ├── data.bak
│ │ ├── data.dat
│ │ ├── data.dir
│ │ └── data.rsc #generated by prep_isce.py
│ ├── 20060924
│ └── ...
├── Igrams
│ ├── 20060624_20061225
│ │ ├── filt_20060624_20061225.cor
│ │ ├── filt_20060624_20061225.cor.vrt
│ │ ├── filt_20060624_20061225.cor.xml
│ │ ├── filt_20060624_20061225_snaphu.unw
│ │ ├── filt_20060624_20061225_snaphu.unw.conncomp
│ │ ├── filt_20060624_20061225_snaphu.unw.conncomp.vrt
│ │ ├── filt_20060624_20061225_snaphu.unw.conncomp.xml
│ │ ├── filt_20060624_20061225_snaphu.unw.vrt
│ │ └──
``` filt_20060624_20061225_snaphu.unw.xml
│ ├── 20060624_20080814
│ └── ...
├── ionosphere
│ ├── 20070107_20071125
│ │ ├── iono.bil.unwCor.filt
│ │ ├── iono.bil.unwCor.filt.vrt
│ │ ├── iono.bil.unwCor.filt.xml
│ │ ├── mask.bil
│ │ ├── mask.bil.vrt
│ │ └── mask.bil.xml
│ ├── 20070107_20080110
│ └── ...
└── mintpy
└── KirishimaAlosAT424F620_630.txt
```
## Configuration File :spiral_note_pad:
```javascript=python
!mkdir m_cfg
```
```javascript=python
cd m_cfg
```
```javascript=python
ls
```
```javascript=python
salton_26515_S2.cfg
```
```javascript=python
!more salton_26515_S2.cfg
```
You can also open the file in a notepad reader, so you be able to edit it or modify it easily.
```javascript=python=
# vim: set filetype=cfg:
##------------------------ smallbaselineApp.cfg ------------------------##
########## computing resource configuration
```
The first step consist in loading data, specifically to get the metadata file from the sensor you are using, this time UAVSAR.
```javascript=python=16
########## 1. load_data
##---------add attributes manually
## MintPy requires attributes listed at: https://mintpy.readthedocs.io/en/latest/api/attributes/
## Missing attributes can be added below manually (uncomment #), e.g.
# ORBIT_DIRECTION = ascending
# PLATFORM = CSK
```
Select the software used to built the stack. :arrow_forward: **i.e.** *isce*
```javascript=python=29
mintpy.load.processor = isce #[isce, aria, hyp3, gmtsar, snap, gamma, roipac], auto for isce
```
On the second step, you can specify the temporal threshold (line 70) you used to built the interferograms stack or use the word `auto` for no, which will consider the implicit dates of the interferograms stack only.
```javascript=python=68
########## 2. modify_network
## 1) Network modification based on temporal/perpendicular baselines, date, num of connections etc.
mintpy.network.tempBaseMax = 180 #[1-inf, no], auto for no, max temporal baseline in days
```
Specify the reference point coordinate (line 112). This could be expressed on **pixel** coordinates or **lat** **long** coordinates.
```javascript=python=104
########## 3. reference_point
## Reference all interferograms to one common point in space
## auto - randomly select a pixel with coherence > minCoherence
## however, manually specify using prior knowledge of the study area is highly recommended
## with the following guideline (section 4.3 in Yunjun et al., 2019):
## 1) located in a coherence area, to minimize the decorrelation effect.
## 2) not affected by strong atmospheric turbulence, i.e. ionospheric streaks
## 3) close to and with similar elevation as the AOI, to minimize the impact of spatially correlated atmospheric delay
mintpy.reference.yx = 455,7680 #[257,151 / auto]
mintpy.reference.lalo = auto #[31.8,130.8 / auto]
mintpy.reference.maskFile = auto #[filename / no], auto for maskConnComp.h5
mintpy.reference.coherenceFile = auto #[filename], auto for avgSpatialCoh.h5
mintpy.reference.minCoherence = auto #[0.0-1.0], auto for 0.85, minimum coherence for auto method
```
For correcting the unwrap error we select **phase** **closure** (line 137).
```javascript=python=129
########## 4. correct_unwrap_error (optional)
## connected components (mintpy.load.connCompFile) are required for this step.
## SNAPHU (Chem & Zebker,2001) is currently the only unwrapper that provides connected components as far as we know.
## reference: Yunjun et al. (2019, section 3)
## supported methods:
## a. phase_closure - suitable for highly redundant network
## b. bridging - suitable for regions separated by narrow decorrelated features, e.g. rivers, narrow water bodies
## c. bridging+phase_closure - recommended when there is a small percentage of errors left after bridging
mintpy.unwrapError.method = phase_closure #[bridging / phase_closure / bridging+phase_closure / no], auto for no
mintpy.unwrapError.waterMaskFile = auto #[waterMask.h5 / no], auto for waterMask.h5 or no [if not found]
```
Specify the minimum temporal coherence for masking (line 179).
```javascript=python=177
## Temporal coherence is calculated and used to generate the mask as the reliability measure
## reference: Pepe & Lanari (2006, IEEE-TGRS)
mintpy.networkInversion.minTempCoh = auto #[0.0-1.0], auto for 0.7, min temporal coherence for mask
mintpy.networkInversion.minNumPixel = auto #[int > 1], auto for 100, min number of pixels in mask above
mintpy.networkInversion.shadowMask = auto #[yes / no], auto for yes [if shadowMask is in geometry file] or no.
```
If you need Solid Earth Tides corrections, specify **yes** (line 194).
```javascript=python=191
########## correct_SET
## Solid Earth tides (SET) correction [need to install insarlab/PySolid]
## reference: Milbert (2018); Fattahi et al. (2020, AGU)
mintpy.solidEarthTides = yes #[yes / no], auto for no
```
* :notebook: You will have several options about tropospheric corrections. select the one better fits to your needs.
> No tropospheric corrections
Specify **no** if you don't need tropospheric corections (line 206).
> ERA5
Specify **pyaps** if you need tropospheric corections (line 206).
Then, it is necessary to specify **auto** or **ERA5** (line 215)
If you specify **auto** (line 216), you must creat a new directory named `WEATHER_DIR/` into the mintpy directory:
```javascript=python
/local3/nacho/UAVSAR/Salton/26515/S02_proc/mintpy
```
```javascript=python
mkdir WEATHER_DIR
```
> GACOS
Specify **gacos** if you need tropospheric corections (line 206).
Then, it is necessary to specify **auto** (line 229). You must creat a new directory named `GACOS/` into the mintpy directory:
```javascript=python
/local3/nacho/UAVSAR/Salton/26515/S02_proc/mintpy
```
```javascript=python
mkdir GACOS
```
* :notebook:You need to manually download GACOS products at http://www.gacos.net for all acquisitions and store them into the new created directory (`./GACOS/`) before running this step.
```javascript=python=197
########## 6. correct_troposphere (optional but recommended)
## correct tropospheric delay using the following methods:
## a. height_correlation - correct stratified tropospheric delay (Doin et al., 2009, J Applied Geop)
## b. pyaps - use Global Atmospheric Models (GAMs) data (Jolivet et al., 2011; 2014)
## ERA5 - ERA5 from ECMWF [need to install PyAPS from GitHub; recommended and turn ON by default]
## MERRA - MERRA-2 from NASA [need to install PyAPS from Caltech/EarthDef]
## NARR - NARR from NOAA [need to install PyAPS from Caltech/EarthDef; recommended for N America]
## c. gacos - use GACOS with the iterative tropospheric decomposition model (Yu et al., 2018a, RSE; 2018b, JGR)
## need to manually download GACOS products at http://www.gacos.net for all acquisitions before running this step
mintpy.troposphericDelay.method = no #[pyaps / height_correlation / gacos / no], auto for pyaps
## Notes for pyaps:
## a. GAM data latency: with the most recent SAR data, there will be GAM data missing, the correction
## will be applied to dates with GAM data available and skipped for the others.
## b. WEATHER_DIR: if you define an environment variable named WEATHER_DIR to contain the path to a
## directory, then MintPy applications will download the GAM files into the indicated directory.
## MintPy application will look for the GAM files in the directory before downloading a new one to
## prevent downloading multiple copies if you work with different dataset that cover the same date/time.
mintpy.troposphericDelay.weatherModel = auto #[ERA5 / MERRA / NARR], auto for ERA5
mintpy.troposphericDelay.weatherDir = auto #[path2directory], auto for WEATHER_DIR or "./"
## Notes for height_correlation:
## Extra multilooking is applied to estimate the empirical phase/elevation ratio ONLY.
## For an dataset with 5 by 15 looks, looks=8 will generate phase with (5*8) by (15*8) looks
## to estimate the empirical parameter; then apply the correction to original phase (with 5 by 15 looks),
## if the phase/elevation correlation is larger than minCorrelation.
mintpy.troposphericDelay.polyOrder = auto #[1 / 2 / 3], auto for 1
mintpy.troposphericDelay.looks = auto #[1-inf], auto for 8, extra multilooking num
mintpy.troposphericDelay.minCorrelation = auto #[0.0-1.0], auto for 0
## Notes for gacos:
## Set the path below to directory that contains the downloaded *.ztd* files
mintpy.troposphericDelay.gacosDir = auto # [path2directory], auto for "./GACOS"
```
Specify if you want to remove phase ramps from each acquisition (line 236-237).
```javascript=python=232
########## 7. deramp (optional)
## Estimate and remove a phase ramp for each acquisition based on the reliable pixels.
## Recommended for localized deformation signals, i.e. volcanic deformation, landslide and land subsidence, etc.
## NOT recommended for long spatial wavelength deformation signals, i.e. co-, post- and inter-seimic deformation.
mintpy.deramp = linear #[no / linear / quadratic], auto for no - no ramp will be removed
mintpy.deramp.maskFile = auto #[filename / no], auto for maskTempCoh.h5, mask file for ramp estimation
```
* :notebook: There are many other options you might consider to configure just like topographic corrections, bootstrapping, geocoding, among others.
Once you have your configuration file ready, next step is running the `smallbaselineApp.py`. Important to check the help option out to do so.
```javascript=python
!smallbaselineApp.py -h
```
```javascript=python
usage: image_stitch.py [-h] -o OUTFILE [--no-offset] [--nodisplay] file1 file2 [file2 ...]
Stitch/mosaic multiple geocoded datasets into one.
The function automatically:
1) finds the common area between adjacent input files
2) calculates the average offset between them
3) apply this average offset to the later file
positional arguments:
file1 file to stitch
file2 file(s) to stitch
optional arguments:
-h, --help show this help message and exit
-o OUTFILE, --output OUTFILE
output file name
--no-offset, --no-off
Do not apply offset if 1) data sets are merely to be stitched AND 2) no adjustment of values needs to be made
(i.e., for two coherence maps), use this flag
--nodisplay do not display the result plotting.
example:
image_stitch.py vel_AlosAT422.h5 vel_AlosAT423.h5 vel_AlosAT424.h5 vel_AlosAT425.h5 -o vel_AlosA.h5
image_stitch.py geom_AlosAT422.h5 geom_AlosAT423.h5 geom_AlosAT424.h5 geom_AlosAT425.h5 -o geom_AlosA.h5 --no-offset
(base) nacho@wireless-ct5:/local3/nacho> smallbaselineApp.py -h
usage: smallbaselineApp.py [-h] [--dir WORKDIR] [-g] [-H] [-v] [--plot] [--start STEP] [--end STEP] [--dostep STEP] [customTemplateFile]
Routine Time Series Analysis for Small Baseline InSAR Stack
positional arguments:
customTemplateFile custom template with option settings.
ignored if the default smallbaselineApp.cfg is input.
optional arguments:
-h, --help show this help message and exit
--dir WORKDIR, --work-dir WORKDIR
work directory, (default: ./).
-g generate default template (if it does not exist) and exit.
-H print the default template file and exit.
-v, --version print software version and exit
--plot plot results [only] without running smallbaselineApp.
steps processing (start/end/dostep):
Command line options for steps processing with names are chosen from the following list:
['load_data', 'modify_network', 'reference_point', 'quick_overview', 'correct_unwrap_error']
['invert_network', 'correct_LOD', 'correct_SET', 'correct_troposphere', 'deramp', 'correct_topography']
['residual_RMS', 'reference_date', 'velocity', 'geocode', 'google_earth', 'hdfeos5']
In order to use either --start or --dostep, it is necessary that a
previous run was done using one of the steps options to process at least
through the step immediately preceding the starting step of the current run.
--start STEP start processing at the named step (default: load_data).
--end STEP, --stop STEP
end processing at the named step (default: hdfeos5)
--dostep STEP run processing at the named step only
reference:
Yunjun, Z., H. Fattahi, and F. Amelung (2019), Small baseline InSAR time series analysis:
Unwrapping error correction and noise reduction, Computers & Geosciences, 133, 104331,
doi:10.1016/j.cageo.2019.104331.
example:
smallbaselineApp.py # run with default template 'smallbaselineApp.cfg'
smallbaselineApp.py <custom_template> # run with default and custom templates
smallbaselineApp.py -h / --help # help
smallbaselineApp.py -H # print default template options
smallbaselineApp.py -g # generate default template if it does not exist
smallbaselineApp.py` -g <custom_template> # generate/update default template based on custom template
smallbaselineApp.py --plot # plot results w/o run [to populate the 'pic' folder after failed runs]
# step processing with --start/stop/dostep options
smallbaselineApp.py GalapagosSenDT128.template --dostep velocity # run at step 'velocity' only
smallbaselineApp.py GalapagosSenDT128.template --end load_data # end after step 'load_data'
```
## Run `smallbaseline.py` :computer:
* Go to the working directory:
```javascript=python
/local3/nacho/UAVSAR/Salton/26515/S02_proc/mintpy
```
* To run this step we write:
```javascript=python
!smallbaselineApp.py ../m_cfg/salton_26515_S2_FSN_SET_GACOS_10img_C0.25b.cfg --end hdfeos5
```
```
/## /## /## /## /#######
| ### /###|__/ | ## | ##__ ##
| #### /#### /## /####### /###### | ## \ ## /## /##
| ## ##/## ##| ##| ##__ ##|_ ##_/ | #######/| ## | ##
| ## ###| ##| ##| ## \ ## | ## | ##____/ | ## | ##
| ##\ # | ##| ##| ## | ## | ## /##| ## | ## | ##
| ## \/ | ##| ##| ## | ## | ####/| ## | #######
|__/ |__/|__/|__/ |__/ \___/ |__/ \____ ##
/## | ##
| ######/
Miami InSAR Time-series software in Python \______/
MintPy 1.5.0.post11, 2022-12-20
---
```
* :notebook:This way we are going to run all the steps on MintPy, from the first one (`load_data`) to the last one (`hdfeos5`).
```javascript=python
Run routine processing with smallbaselineApp.py on steps: ['load_data', 'modify_network', 'reference_point', 'quick_overview', 'correct_unwrap_error', 'invert_network', 'correct_LOD', 'correct_SET', 'correct_troposphere', 'deramp', 'correct_topography', 'residual_RMS', 'reference_date', 'velocity', 'geocode', 'google_earth', 'hdfeos5']
```
The message you'll see at the end of the processing is:
```javascript=python=1742
Go back to directory: /local3/nacho/UAVSAR/Salton/26515/S02_Proc/mintpy
################################################
Normal end of smallbaselineApp processing!
################################################
Time used: 247 mins 10.2 secs
```
The line number and time used might vary in function of the amount of data considered in the process.
[Reference](https://github.com/forrestfwilliams/UAVSAR_InSAR/blob/main/mintpy_processing_0550201_01.ipynb)
> [TOC]
Sign in with Wallet
Connect another wallet