[TOC]
# Datacube Importing Service
## 功能需求
- 將Sentinel跟Landsat影像透過自動抓取的方式更新到Datacube(Rasdaman)
## 使用工具
<details>
<summary>原廠提供匯入工具支援Sentinel的影像,如S1-GRD, S2-L2A, S3-OLCI, S5P以及另外的大氣分析數據ERA5共5種, Landsat必須由我們再擴展功能才能支援</summary>
<summary>S1-GRD</summary>
S1: 哨兵1號(Sentinel-1)衛星,是歐空局哥白尼計劃的一部分。它主要用於全天候、全天時的雷達成像。
GRD: Ground Range Detected,地面距離檢測產品。這是經過處理後的雷達影像,具有一定的幾何校正和放射校正。
哨兵1號衛星的主要特點:
使用C波段合成孔徑雷達(SAR)。
提供高分辨率的雷達影像。
主要用於地表變形監測、海洋監測、冰川和極地冰蓋監測等。
<summary>S2-L2A</summary>
S2: 哨兵2號(Sentinel-2)衛星,是歐空局哥白尼計劃的一部分。它主要用於陸地和沿海區域的光學成像。
L2A: Level-2A產品,表示已進行大氣校正的地表反射率數據。
哨兵2號衛星的主要特點:
提供高分辨率多光譜影像(10米至60米分辨率)。
覆蓋13個光譜波段。
主要用於農業、森林監測、水體監測、土地覆蓋和災害管理等應用。
<summary>S3-OLCI</summary>
S3: 哨兵3號(Sentinel-3)衛星,是歐空局哥白尼計劃的一部分。它旨在提供海洋和陸地表面監測數據。
OLCI: Ocean and Land Colour Instrument,海洋和陸地顏色儀器,是哨兵3號上的一個光學成像儀器,用於測量海洋和陸地表面的顏色。
哨兵3號衛星和OLCI儀器的主要特點:
提供寬覆蓋範圍(寬達1270公里的成像寬度)。
覆蓋21個光譜波段。
主要用於海洋水質監測、陸地植被狀態監測、大氣成分分析等。
<summary>S5P</summary>
S5P: 哨兵5P(Sentinel-5 Precursor)衛星,主要用於監測大氣成分,特別是氣體污染物。
TROPOMI: Tropospheric Monitoring Instrument,對流層監測儀器,是S5P上的主要儀器。
哨兵5P衛星的主要特點:
監測對流層中的各種氣體,如臭氧、二氧化氮、二氧化硫、甲烷和一氧化碳。
用於空氣質量監測、氣候研究和健康影響分析。
<summary>ERA5</summary>
ERA5: 由歐洲中期天氣預報中心(ECMWF)提供的第五代大氣再分析數據集。
再分析數據集通過將歷史天氣觀測數據與數值天氣預報模型相結合,提供了一個一致的全球氣象記錄。
ERA5的主要特點:
提供高分辨率的全球氣象數據(水平分辨率約為31公里)。
時間分辨率高(每小時一次的數據)。
包含大量氣象變量,如溫度、降水、風速、濕度等。
用於氣候監測、天氣預報、環境研究和風險管理等。
</details>
## 影像來源
- https://dataspace.copernicus.eu/
- 提供多種地球觀測數據(Earth Observation Data, EOData)
- 
- 有申請帳號介接API
## 介接狀況
- 目前實作介接S1-GRD的資料, 可以將資料從平台上下載至Rasdaman, 但後半段的匯入動作, 沒有執行成功
- 實際匯入時的command log
```shell=
administrator@ubuntu-122-80:~/gisfcu/tools$ ./dataservice_import.sh ./profile/example_profile.toml ./log.txt
dataservice_import.sh: all output is also written to ./log.txt
Parsed profile:
Service: rasdias
Products: {'s1-grd': {'cov_id': 'S1_GRDH_IW_HH', 'cov_id_regex_pattern': 'S1_GRD._[EIS].*', 'cov_id_suffix': '', 'list_files': True, 'preprocess_files': True, 'downscale_files': True, 'import_pyramids': True, 'import_base_coverages': True, 'generate_ingredients': True, 'upload_files': False, 'upload_hosts': ['user@host'], 'delete_files': False, 'import_timeout': 1800, 'overviewsdir': '/home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/', 'external_overview_levels': [2, 3, 4, 5, 6, 7], 'nested_pyramid_levels': [], 'vrtdir': '/home/administrator/gisfcu/tools/data/geo/Sentinel-1/grd/', 'resolution_iw': '0.0001', 'resolution_ew_h': '0.0005', 'resolution_ew_m': '0.001', 'resolution_sm': '0.0001', 'modebeams': ['IW'], 'polarisations': ['HH'], 'from_date': '2015-01-30', 'to_date': '2015-01-30'}}
Workdir: /home/administrator/gisfcu/tools/data/import/
Rmanhome: /opt/rasdaman/
Petascope: http://localhost:8080/rasdaman/ows
Creds: /home/administrator/gisfcu/tools/cfg/.rasadmin_credentials
User: rasadmin
Dry run: False
Hostname: ubuntu-122-80
################################################################################
Updating s1-grd datacubes
################################################################################
Import will start from the configured from_date 2015-01-30.
Import will end with the configured to_date 2015-01-30.
removing existing directory /home/administrator/gisfcu/tools/data/import/s1-grd/filepaths/
removing existing directory /home/administrator/gisfcu/tools/data/import/s1-grd/ingredients/
removing existing directory /home/administrator/gisfcu/tools/data/import/s1-grd/tmp/
================================================================================
Importing date 2015-01-30 ...
================================================================================
2015-01-30 Starting: list files in /home/administrator/gisfcu/tools/data/import/s1-grd/filepaths/2015_01_30.txt ...
catalog search criteria: {'productType': 'S1_SAR_GRD', 'start': '2015-01-30', 'end': '2015-01-31', 'items_per_page': 1000, 'geom': {'lonmin': -7.876046, 'latmin': 36.78478, 'lonmax': -1.171323, 'latmax': 43.270077}}
catalog returned 3 scenes.
removing duplicates resulted in 3 scenes.
filtering resulted in 3 scenes.
processing resulted in 3 scenes.
filtering resulted in 3 scenes.
2015-01-30 Finished: list 6 files of total size 5.4 GB in /home/administrator/gisfcu/tools/data/import/s1-grd/filepaths/2015_01_30.txt in 5.73 s.
2015-01-30 Starting: generate VRT files in /home/administrator/gisfcu/tools/data/geo/Sentinel-1/grd/2015/01/30/ ...
Total size of VRTs: 360.7 KB
2015-01-30 Finished: generate VRT files in /home/administrator/gisfcu/tools/data/geo/Sentinel-1/grd/2015/01/30/ in 0.17 s.
--------------------------------------------------------------------------------
2015-01-30 Starting: create overviews in /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/ ...
Collected 11 VRT files to downscale.
Downscaling VRTs in parallel with 4 processes.
Downloaded in 1.47 m: s3://eodata/Sentinel-1/SAR/GRD/2015/01/30/S1A_IW_GRDH_1SDV_20150130T064145_20150130T064210_004397_0055F0_7CA0.SAFE/measurement/s1a-iw-grd-vh-20150130t064145-20150130t064210-004397-0055f0-002.tiff
Downloaded in 1.47 m: s3://eodata/Sentinel-1/SAR/GRD/2015/01/30/S1A_IW_GRDH_1SDV_20150130T050304_20150130T050329_004396_0055E9_BF26.SAFE/measurement/s1a-iw-grd-vh-20150130t050304-20150130t050329-004396-0055e9-002.tiff
Downloaded in 1.48 m: s3://eodata/Sentinel-1/SAR/GRD/2015/01/30/S1A_IW_GRDH_1SDV_20150130T064210_20150130T064235_004397_0055F0_07C8.SAFE/measurement/s1a-iw-grd-vh-20150130t064210-20150130t064235-004397-0055f0-002.tiff
Downloaded in 1.64 m: s3://eodata/Sentinel-1/SAR/GRD/2015/01/30/S1A_IW_GRDH_1SDV_20150130T064116_20150130T064145_004397_0055F0_512D.SAFE/measurement/s1a-iw-grd-vh-20150130t064116-20150130t064145-004397-0055f0-002.tiff
1 / 11 - top pyramid level of size 6.9 MB generated in 1.83 m: /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/S1_GRDH_IW_DV_VH/s1a-iw-grd-vh-20150130t050304-20150130t050329-004396-0055e9-002_GRDH_DV_warp.tif_8
1 / 11 - downscaled to 5 levels in 1.41 s
2 / 11 - top pyramid level of size 8.8 MB generated in 2.09 m: /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/S1_GRDH_IW_DV_VH/s1a-iw-grd-vh-20150130t064116-20150130t064145-004397-0055f0-002_GRDH_DV_warp.tif_8
2 / 11 - downscaled to 5 levels in 0.96 s
3 / 11 - top pyramid level of size 7.4 MB generated in 1.83 m: /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/S1_GRDH_IW_DV_VH/s1a-iw-grd-vh-20150130t064145-20150130t064210-004397-0055f0-002_GRDH_DV_warp.tif_8
3 / 11 - downscaled to 5 levels in 0.91 s
4 / 11 - top pyramid level of size 6.8 MB generated in 1.82 m: /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/S1_GRDH_IW_DV_VH/s1a-iw-grd-vh-20150130t064210-20150130t064235-004397-0055f0-002_GRDH_DV_warp.tif_8
4 / 11 - downscaled to 5 levels in 0.91 s
Downloaded in 1.45 m: s3://eodata/Sentinel-1/SAR/GRD/2015/01/30/S1A_IW_GRDH_1SDV_20150130T032440_20150130T032505_004395_0055E2_D768.SAFE/measurement/s1a-iw-grd-vv-20150130t032440-20150130t032505-004395-0055e2-001.tiff
Downloaded in 1.45 m: s3://eodata/Sentinel-1/SAR/GRD/2015/01/30/S1A_IW_GRDH_1SDV_20150130T050304_20150130T050329_004396_0055E9_BF26.SAFE/measurement/s1a-iw-grd-vv-20150130t050304-20150130t050329-004396-0055e9-001.tiff
Downloaded in 1.71 m: s3://eodata/Sentinel-1/SAR/GRD/2015/01/30/S1A_IW_GRDH_1SDV_20150130T174416_20150130T174445_004404_005617_40DE.SAFE/measurement/s1a-iw-grd-vh-20150130t174416-20150130t174445-004404-005617-002.tiff
Downloaded in 1.67 m: s3://eodata/Sentinel-1/SAR/GRD/2015/01/30/S1A_IW_GRDH_1SDV_20150130T064116_20150130T064145_004397_0055F0_512D.SAFE/measurement/s1a-iw-grd-vv-20150130t064116-20150130t064145-004397-0055f0-001.tiff
5 / 11 - top pyramid level of size 7.5 MB generated in 2.11 m: /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/S1_GRDH_IW_DV_VH/s1a-iw-grd-vh-20150130t174416-20150130t174445-004404-005617-002_GRDH_DV_warp.tif_8
5 / 11 - downscaled to 5 levels in 0.88 s
6 / 11 - top pyramid level of size 8.9 MB generated in 1.83 m: /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/S1_GRDH_IW_DV_VV/s1a-iw-grd-vv-20150130t032440-20150130t032505-004395-0055e2-001_GRDH_DV_warp.tif_8
6 / 11 - downscaled to 5 levels in 0.93 s
7 / 11 - top pyramid level of size 8.9 MB generated in 1.80 m: /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/S1_GRDH_IW_DV_VV/s1a-iw-grd-vv-20150130t050304-20150130t050329-004396-0055e9-001_GRDH_DV_warp.tif_8
7 / 11 - downscaled to 5 levels in 0.93 s
8 / 11 - top pyramid level of size 12.3 MB generated in 2.12 m: /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/S1_GRDH_IW_DV_VV/s1a-iw-grd-vv-20150130t064116-20150130t064145-004397-0055f0-001_GRDH_DV_warp.tif_8
8 / 11 - downscaled to 5 levels in 1.04 s
Downloaded in 1.41 m: s3://eodata/Sentinel-1/SAR/GRD/2015/01/30/S1A_IW_GRDH_1SDV_20150130T064145_20150130T064210_004397_0055F0_7CA0.SAFE/measurement/s1a-iw-grd-vv-20150130t064145-20150130t064210-004397-0055f0-001.tiff
Downloaded in 1.44 m: s3://eodata/Sentinel-1/SAR/GRD/2015/01/30/S1A_IW_GRDH_1SDV_20150130T064210_20150130T064235_004397_0055F0_07C8.SAFE/measurement/s1a-iw-grd-vv-20150130t064210-20150130t064235-004397-0055f0-001.tiff
9 / 11 - top pyramid level of size 9.6 MB generated in 1.76 m: /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/S1_GRDH_IW_DV_VV/s1a-iw-grd-vv-20150130t064145-20150130t064210-004397-0055f0-001_GRDH_DV_warp.tif_8
9 / 11 - downscaled to 5 levels in 0.94 s
10 / 11 - top pyramid level of size 9.2 MB generated in 1.77 m: /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/S1_GRDH_IW_DV_VV/s1a-iw-grd-vv-20150130t064210-20150130t064235-004397-0055f0-001_GRDH_DV_warp.tif_8
10 / 11 - downscaled to 5 levels in 0.95 s
Downloaded in 1.67 m: s3://eodata/Sentinel-1/SAR/GRD/2015/01/30/S1A_IW_GRDH_1SDV_20150130T174416_20150130T174445_004404_005617_40DE.SAFE/measurement/s1a-iw-grd-vv-20150130t174416-20150130t174445-004404-005617-001.tiff
11 / 11 - top pyramid level of size 9.3 MB generated in 2.07 m: /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/S1_GRDH_IW_DV_VV/s1a-iw-grd-vv-20150130t174416-20150130t174445-004404-005617-001_GRDH_DV_warp.tif_8
11 / 11 - downscaled to 5 levels in 0.96 s
Total size of downscaled files: 127.5 MB
2015-01-30 Finished: create overviews in /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/ in 6.02 m.
--------------------------------------------------------------------------------
2015-01-30 Starting: merge overviews in /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/ ...
Merged 6 files into /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/S1_GRDH_IW_DV_VV/8/merged.tif in 3.35 s, removing input files.
Retiled /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/S1_GRDH_IW_DV_VV/8/merged.tif in 11.59 s, removing it.
Merged 5 files into /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/S1_GRDH_IW_DV_VH/8/merged.tif in 2.86 s, removing input files.
Retiled /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/S1_GRDH_IW_DV_VH/8/merged.tif in 7.56 s, removing it.
Merged 6 files into /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/S1_GRDH_IW_DV_VV/16/merged.tif in 0.98 s, removing input files.
Retiled /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/S1_GRDH_IW_DV_VV/16/merged.tif in 3.48 s, removing it.
Merged 5 files into /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/S1_GRDH_IW_DV_VH/16/merged.tif in 0.88 s, removing input files.
Retiled /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/S1_GRDH_IW_DV_VH/16/merged.tif in 2.10 s, removing it.
Merged 6 files into /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/S1_GRDH_IW_DV_VV/32/merged.tif in 0.42 s, removing input files.
Retiled /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/S1_GRDH_IW_DV_VV/32/merged.tif in 1.02 s, removing it.
Merged 5 files into /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/S1_GRDH_IW_DV_VH/32/merged.tif in 0.38 s, removing input files.
Retiled /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/S1_GRDH_IW_DV_VH/32/merged.tif in 0.67 s, removing it.
Merged 6 files into /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/S1_GRDH_IW_DV_VV/64/merged.tif in 0.28 s, removing input files.
Retiled /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/S1_GRDH_IW_DV_VV/64/merged.tif in 0.35 s, removing it.
Merged 5 files into /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/S1_GRDH_IW_DV_VH/64/merged.tif in 0.25 s, removing input files.
Retiled /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/S1_GRDH_IW_DV_VH/64/merged.tif in 0.25 s, removing it.
Merged 6 files into /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/S1_GRDH_IW_DV_VV/128/merged.tif in 0.22 s, removing input files.
Retiled /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/S1_GRDH_IW_DV_VV/128/merged.tif in 0.16 s, removing it.
Merged 5 files into /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/S1_GRDH_IW_DV_VH/128/merged.tif in 0.20 s, removing input files.
Retiled /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/S1_GRDH_IW_DV_VH/128/merged.tif in 0.14 s, removing it.
Merged 6 files into /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/S1_GRDH_IW_DV_VV/256/merged.tif in 0.20 s, removing input files.
Retiled /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/S1_GRDH_IW_DV_VV/256/merged.tif in 0.12 s, removing it.
Merged 5 files into /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/S1_GRDH_IW_DV_VH/256/merged.tif in 0.21 s, removing input files.
Retiled /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/S1_GRDH_IW_DV_VH/256/merged.tif in 0.10 s, removing it.
Total size of merged overviews: 128.7 MB
2015-01-30 Finished: merge overviews in /home/administrator/gisfcu/tools/data_big/geo/Sentinel-1/GRD/overviews/2015/01/30/ in 37.91 s.
--------------------------------------------------------------------------------
2015-01-30 Starting: generate all ingredients ...
2015-01-30 Starting: collect filepaths /home/administrator/gisfcu/tools/data/import/s1-grd/filepaths_warped/2015-01-30_GRDH_DH_HH.txt ...
Empty filepaths /home/administrator/gisfcu/tools/data/import/s1-grd/filepaths_warped/2015-01-30_GRDH_DH_HH.txt, skipping.
2015-01-30 Starting: collect filepaths /home/administrator/gisfcu/tools/data/import/s1-grd/filepaths_warped/2015-01-30_GRDM_DH_HH.txt ...
Empty filepaths /home/administrator/gisfcu/tools/data/import/s1-grd/filepaths_warped/2015-01-30_GRDM_DH_HH.txt, skipping.
2015-01-30 Starting: collect filepaths /home/administrator/gisfcu/tools/data/import/s1-grd/filepaths_warped/2015-01-30_GRDH_DV_HH.txt ...
Empty filepaths /home/administrator/gisfcu/tools/data/import/s1-grd/filepaths_warped/2015-01-30_GRDH_DV_HH.txt, skipping.
2015-01-30 Starting: collect filepaths /home/administrator/gisfcu/tools/data/import/s1-grd/filepaths_warped/2015-01-30_GRDM_DV_HH.txt ...
Empty filepaths /home/administrator/gisfcu/tools/data/import/s1-grd/filepaths_warped/2015-01-30_GRDM_DV_HH.txt, skipping.
2015-01-30 Finished: generate all ingredients in 0.01 s.
No ingredient files matching *.json in /home/administrator/gisfcu/tools/data/import/s1-grd/ingredients/ found to import.
================================================================================
Importing date 2015-01-30 done in 6.75 m.
================================================================================
Clearing temporary files...
No temporary files to remove.
================================================================================
Updated s1-grd datacubes in 6.75 m.
```
- 11個VRT檔案截圖

- downscale截圖

- 準備好的tiff準備匯入

- 後續處理:
1.為何後續的匯入流程沒有執行
2.繼續將討論後要匯入的影像實作自動匯入, 實務上必須by不同得影像類型去實作

## 討論事項
- 需要先匯入那些影像?
- 時間?頻率?
- 影像範圍?
- 空間?放哪台
- 有太多條件可以下了
- SATELLITE PLATFORM
- Sentinel-1A (operational since 2014) and Sentinel-1B (operational from 2016 to 2022
- ORBIT DIRECTION
- 下降軌道(飛行方向:北 - 南)還是上升軌道
- RELATIVE ORBIT NUMBER
- 軌道號。Sentinel-1衛星處於接近極地的太陽同步軌道,重複周期為12天,每個周期包含175個軌道(針對單顆衛星)。
- ACQUISITION MODE
- Stripmap (SM): 條帶模式
Interferometric Wide swath (IW): 干涉寬幅模式
Extra-Wide swath (EW): 超寬幅模式
Wave (WV): 波浪模式
- BEAM ID
- S1-S6 用於條帶模式 (SM 模式)
IW1-IW3 用於干涉寬幅模式 (IW 模式)
EW1-EW5 用於超寬幅模式 (EW 模式)
WV1-WV2 用於波浪模式 (WV 模式)
- POLARISATION
- HH 極化: 例如,在城市區域監測中,可以識別並分析建築物和其他金屬結構。
VV 極化: 例如,在農業監測中,可以觀察農田和植被的生長狀況。
VV+VH 極化: 例如,在混合地表監測中,可以更好地區分和識別不同地表特徵,如森林中的不同樹種。
HH+HV 極化: 例如,在農業和森林監測中,可以更全面地了解地表結構和材料的多樣性。
- PRODUCT AVAILABILITY
- 都選Immediate, 沒有要下單訂購
