ALTIUS Secondary Product

• ALTIUS Secondary Product
  • Product Format Definition
    • Catalogue Collections
    • Product Types and File name
  • List of Component Updates
  • List of Documents Updates

Product Format Definition

Catalogue Collections

The identifiers for the logical collections (grouping multiple product types listed in section 4.2.1) currently expected are:

• ALTIUS_L1C
• ALTIUS_L2_O3_NRT
• ALTIUS_L2_O3_CP

The syntax is: ALTIUS_<LL>_<Species>_<Timeliness>, where the variables are defined as follows:
* <Species>: the product species (O3, NO2, AER, MSO, NO3, H2O, TMP, BRO OCL)
* <LL>: the product level (L0, L1, or L2)
* <Timeliness>: the product timeliness (NRT, CP, or nothing if not applicable)

[CNL] From the rule above, I have derived the collection names for secondary product:

• ALTIUS_NO2_CP
• ALTIUS_AER_CP
• ALTIUS_MSO_CP
• ALTIUS_NO3_CP
• ALTIUS_H2O_CP
• ALTIUS_TMP_CP
• ALTIUS_BRO_CP
• ALTIUS_OCL_CP

Product Types and File name

Ozone Products

List of product types (= file type):

• ____BL__1C
• ____SOO_1C
• ____STO_1C
• NO3_BL__2_
• NO3_SOO_2_
• NO3_STO_2_
• CO3_BL__2_
• CO3_SOO_2_
• CO3_STO_2_

Not exposed (L0):

• ___BL__0
• ___SOO_0
• ___STO_0
• ___CAL_0

Seems to be ignored (not exposed) probably because was meant initally for secondary product or synthesis products

• _______0
• ________1C
• _______S1C
• CO3_____2_
• CO3____S2_
• NO3_____2_
• NO3____S2_

Secondary Products

The naming convention used for the primary product O3 allows for supporting other products. The secondary products naming convention shall therefore follow the same principle. The secondary products are generated only through consolidated product processing, the O3 primary product being prioritized for NRT processing. The set of observation geometries retained for profile retrieval depends on the considered secondary species. Table 10 presents the file name convention for the secondary species and their associated trigram

Secondary Products Types

• CNO2BL__2_
• CNO2STO_2_
• CNO2SOO_2_
• CAERBL__2_
• CAERSTO_2_
• CAERSOO_2_
• CMSOSTO_2_
• CMSOSOO_2_
• CNO3STO_2_
• CNO3SOO_2_
• CH2OBL__2_
• CH2OSOO_2_
• CTMPBL__2_
• CTMPSOO_2_
• CBROSOO_2_
• COCLSOO_2_

List of Component Updates

1. Catalogue > Gery N currently working in this section

   • root
     • compose/catalogue/dataset-md-import/conf/metalink.meta4 (docker-compose configuration)
       Partial content of file:

       <?xml version="1.0" encoding="UTF-8"?> <metalink xmlns="urn:ietf:params:xml:ns:metalink"> <published>2022-05-30T13:00:00Z</published> <file> <identity>ALTIUS_L1C</identity> <version>1.0</version> <description>Directory with index files for ALTIUS_L1C collection to be ingested by FedEO</description> <url>http://dissemination:8080/oads/meta/____BL__1C/index</url> <metaurl mediatype="application/xml">configuration-oads.xml</metaurl> </file>

     • compose/clean.sh (script used to clean docker-compose resrouces)
       Content of script:

       rm -rf ./solr/config/cat/data rm -rf ./solr/config/dataset/data rm -rf ./solr/config/films/data rm -rf ./dataset-md-import/working rm -rf ./nas/store/products/TEST_____BL__1C rm -rf ./nas/store/products/TEST_____SOO_1C rm -rf ./nas/store/products/TEST____STO_1C/2022/04 rm -rf ./nas/store/products/TEST_CO3_BL__2_ rm -rf ./nas/store/products/TEST_CO3_SOO_2_ rm -rf ./nas/store/products/TEST_CO3_STO_2_ rm -rf ./nas/store/products/TEST_NO3_BL__2_ rm -rf ./nas/store/products/TEST_NO3_SOO_2_ rm -rf ./nas/store/products/TEST_NO3_STO_2_ rm -rf ./nas/dissemination

     • compose/series-md-im-export/localCollections/altius (xml files - 1 per collection, add new files for new collections)

       <?xml version="1.0" encoding="UTF-8"?> <gmi:MI_Metadata xmlns:gmi="http://www.isotc211.org/2005/gmi" xmlns:gco="http://www.isotc211.org/2005/gco" xmlns:gmd="http://www.isotc211.org/2005/gmd" xmlns:gml="http://www.opengis.net/gml/3.2" xmlns:gmx="http://www.isotc211.org/2005/gmx" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.isotc211.org/2005/gmi file://HALLEY/bu-user-segment/projects/ceos-2019/Deliverables/D2.2%20-%20Service%20Discovery%20White%20Paper/iso-validation-schemas/inspire/gmi/gmi.xsd"> <gmd:fileIdentifier> <gco:CharacterString>TEST_ALTIUS_L1C</gco:CharacterString> </gmd:fileIdentifier>.....

     • postman.json
       Requests used to generate products (in postman collection), examples:

       ... "request": { "method": "GET", "header": [], "url": { "raw": "{{dmUrl}}/simulator?groupId=TEST_NO3_BL__2_&size=10&startDate=2022-05-01T00:00:00Z&dissemination=true", ...

   • dataset-md-import
     • conf/metalink.meta4
       Partial content of file:

       <?xml version="1.0" encoding="UTF-8"?> <metalink xmlns="urn:ietf:params:xml:ns:metalink"> <published>2022-05-30T13:00:00Z</published> <file> <identity>ALTIUS_L1C</identity> <version>1.0</version> <description>Directory with index files for ALTIUS_L1C collection to be ingested by FedEO</description> <url>http://dissemination:8080/oads/meta/____BL__1C/index</url> <metaurl mediatype="application/xml">configuration-oads.xml</metaurl> </file>

   • series-md-export
     • series-md-im-export/localCollections/altius (xml files - 1 per collection, add new files for new collections)
       • Current
         • TEST_ALTIUS_L1C.xml
         • TEST_ALTIUS_L2_O3_CP.xml
         • TEST_ALTIUS_L2_O3_NRT.xml
       • NEW
         • ALTIUS_AER_CP.xml
         • ALTIUS_BRO_CP.xml
         • ALTIUS_H2O_CP.xml
         • ALTIUS_MSO_CP.xml
         • ALTIUS_NO2_CP.xml
         • ALTIUS_NO3_CP.xml
         • ALTIUS_OCL_CP.xml
         • ALTIUS_TMP_CP.xml
         • TEST_ALTIUS_L1C.xml
         • TEST_ALTIUS_L2_O3_CP.xml
         • TEST_ALTIUS_L2_O3_NRT.xml

2. Dissemination

   • dissemination-ftp

     • resources/META-INF/microprofile.properties or k8s/dissemination_configmap.yaml
       • ftp.servers: 'collections' property of ftp servers
         Note: understand why id formatted as ALTIUS.BL_1A (where does it come from ?)
         K8S configmap value:
         FTP_SERVERS: "[{\"id\":\"ftp1\",\"server\":\"sftp-server\",\"username\":\"user\",\"password\":\"pwd\",\"port\":22,\"directory\":\"content/dissemination\",\"retries\":3,\"collections\":[\"ALTIUS.BL_0\",\"ALTIUS.BL_1A\",\"ALTIUS.BL_1B\",\"ALTIUS.BL_1C\",\"ALTIUS.NRT_BL__2\",\"ALTIUS.NRT_STO_2\",\"ALTIUS.SOO_0\",\"ALTIUS.STO_0\",\"ALTIUS.STO_1A\",\"ALTIUS.STO_1B\"]}]"
         NEW
         FTP_SERVERS: '[{\"id\":\"ftp1\",\"server\":\"sftp-server\",\"username\":\"user\",\"password\":\"pwd\",\"port\":22,\"directory\":\"content/dissemination\",\"retries\":3,\"collections\":[\"ALTIUS.BL_0\",\"ALTIUS.BL_1A\",\"ALTIUS.BL_1B\",\"ALTIUS.BL_1C\",\"ALTIUS.NRT_BL__2\",\"ALTIUS.NRT_STO_2\",\"ALTIUS.SOO_0\",\"ALTIUS.STO_0\",\"ALTIUS.STO_1A\",\"ALTIUS.STO_1B\",\"CNO2BL__2_\",\"CNO2STO_2_\",\"CNO2SOO_2_\",\"CAERBL__2_\",\"CAERSTO_2_\",\"CAERSOO_2_\",\"CMSOSTO_2_\",\"CMSOSOO_2_\",\"CNO3STO_2_\",\"CNO3SOO_2_\",\"CH2OBL__2_\",\"CH2OSOO_2_\",\"CTMPBL__2_\",\"CTMPSOO_2_\",\"CBROSOO_2_\",\"COCLSOO_2_\"]}]'

   • dissemination-indexer

     • resources/META-INF/microprofile.properties or k8s/dissemination_configmap.yaml
       • oads.filters
         Current K8S configmap value:
         OADS_FILTERS: "{\"NO3_BL__2_\":{},\"NO3_SOO_2_\":{},\"NO3_STO_2_\":{},\"CO3_BL__2_\":{},\"CO3_SOO_2_\":{},\"CO3_STO_2_\":{},\"____STO_1C\":{},\"____SOO_1C\":{},\"____BL__1C\":{}}"
         New config:
         OADS_FILTERS: '{\"NO3_BL__2_\":{},\"NO3_SOO_2_\":{},\"NO3_STO_2_\":{},\"CO3_BL__2_\":{},\"CO3_SOO_2_\":{},\"CO3_STO_2_\":{},\"____STO_1C\":{},\"____SOO_1C\":{},\"____BL__1C\":{},\"CNO2BL__2_\":{},\"CNO2STO_2_\":{},\"CNO2SOO_2_\":{},\"CAERBL__2_\":{},\"CAERSTO_2_\":{},\"CAERSOO_2_\":{},\"CMSOSTO_2_\":{},\"CMSOSOO_2_\":{},\"CNO3STO_2_\":{},\"CNO3SOO_2_\":{},\"CH2OBL__2_\":{},\"CH2OSOO_2_\":{},\"CTMPBL__2_\":{},\"CTMPSOO_2_\":{},\"CBROSOO_2_\":{},\"COCLSOO_2_\":{}}'

   • dissemination-web

     • resources/META-INF/microprofile.properties or k8s/dissemination_configmap.yaml
       • oads.filters
         K8S configmap value:
         OADS_FILTERS: "{\"NO3_BL__2_\":{},\"NO3_SOO_2_\":{},\"NO3_STO_2_\":{},\"CO3_BL__2_\":{},\"CO3_SOO_2_\":{},\"CO3_STO_2_\":{},\"____STO_1C\":{},\"____SOO_1C\":{},\"____BL__1C\":{}}"
         New config:
         OADS_FILTERS: '{\"NO3_BL__2_\":{},\"NO3_SOO_2_\":{},\"NO3_STO_2_\":{},\"CO3_BL__2_\":{},\"CO3_SOO_2_\":{},\"CO3_STO_2_\":{},\"____STO_1C\":{},\"____SOO_1C\":{},\"____BL__1C\":{},\"CNO2BL__2_\":{},\"CNO2STO_2_\":{},\"CNO2SOO_2_\":{},\"CAERBL__2_\":{},\"CAERSTO_2_\":{},\"CAERSOO_2_\":{},\"CMSOSTO_2_\":{},\"CMSOSOO_2_\":{},\"CNO3STO_2_\":{},\"CNO3SOO_2_\":{},\"CH2OBL__2_\":{},\"CH2OSOO_2_\":{},\"CTMPBL__2_\":{},\"CTMPSOO_2_\":{},\"CBROSOO_2_\":{},\"COCLSOO_2_\":{}}'

3. Data Store:

   • Web UI: implmement filter and column species, optimize level & baseline width
   • Add urls for creating secondary products: resources/validation/simulator-urls.txt
   • update: resources/integration/import.sql
   • update: resources/integration/products.md
   • udate postman collection: (resources/integration/postman.json)

4. Portal (catalogue UI)

   • Rename "Illumination Azimuth Angle" to "Illumniation Azimuth" (same for Zenith)

5. LTDP (Long Term Data Preservation)

   • data-preservation-server
     • resources/META-INF/microprofile.properties or k8s/ltdp_configmap.yaml
       • collections
         Current K8S configmap value:
         COLLECTIONS: CO3_BL__2_,CO3_SOO_2_,CO3_STO_2_,NO3_BL__2_,NO3_SOO_2_,NO3_STO_2_,____BL__1C,____SOO_1C,____STO_1C
         NEW
         COLLECTIONS: CO3_BL__2_,CO3_SOO_2_,CO3_STO_2_,NO3_BL__2_,NO3_SOO_2_,NO3_STO_2_,____BL__1C,____SOO_1C,____STO_1C,CNO2BL__2_,CNO2STO_2_,CNO2SOO_2_,CAERBL__2_,CAERSTO_2_,CAERSOO_2_,CMSOSTO_2_,CMSOSOO_2_,CNO3STO_2_,CNO3SOO_2_,CH2OBL__2_,CH2OSOO_2_,CTMPBL__2_,CTMPSOO_2_,CBROSOO_2_,COCLSOO_2_

List of Documents Updates

CNLcurrently working in this section

1. Product Format Definition
   • Refined parentIdentifier definition
   • Made some comments for consistency
2. External interfaces SDD: /
3. Internal ICD:
   • Adjusted the definition of the product group (product type metadata property)
4. Data Distribution SDD: