---
title: 'Description of the IPRAL data flow scheme applied for SCC data processing'
disqus: SCC
---

Description of the IPRAL data flow scheme applied for SCC data processing
===

## Table of Contents

[TOC]

## Description of the requirements

Tools already or to be implemented
---

> Raw2scc [name=MAD]

Feature: extract RAW licel files and sort them
  Scenario: Unzip repository in BDD sirta and get the RM licel files
    Given raw2scc access the zip files, unzip and read RM files
    Then raw2scc saves the RM files for one haour range HH-30mn,HH+30mn

> licel2scc [name=MAD][name=ioannis][name=CP]

Feature: convert raw RM licel files to netcdf files
  nc files compatible with SCC algo
  uses IPRAL systems configuration
  Scenario: licel2scc reads RM licel files and convert to nc files
    Given IPRAL configuration py file containing related channel list of IPRAL in RM files and channel list in SCC interface
    And convert
    Then provide one nf file per hour of measurements

> scc_access API [name=MAD][name=ioannis]

Feature: submit nc input files to SCC algo using API interface
  Scenario: interact with SCC web interface
    Given a nc input file
    And a system number of the SCC configuration for IPRAL lidar
    Then SCC algo process the corresponding measurement of the nc file
    And returns the results of the submission process (success or fail)
    And returnes the products out of SCC processing or the error messages from SCC

> Single Calculus Chain Configuration [name=CP][name=Giuseppe]

Feature: Channels configuration and Processing scheme
  describe IPRAL channels used for optical retrievals
  describe IPRAL product retrievals depending of system
  take into accound telescope and if raman channels are on or off
  Scenario: submit nec files linked to a system number
    Given system number
    Then apply the corresponding processing schemes for that system
    And provide nc products files for each stage of retrievals
    And error messages obtained during the processing

Diagram
---

%%{init: {'theme':'forest'}}%%
graph TB


linkStyle default color:#9f6,fill:none,stroke:black,stroke-width:2px;
  raw_prep --> climserv --> SubGraph1Flow
  
  subgraph "SIRTA processing"
     climserv[(climserv server)]
     SubGraph1Flow(Read RM headers <br/> field BT3 <br/> 9th column)
     SubGraph1Flow -- 0 --> DoChoice1[Raman included] --> raw2scc{raw2scc} 
     SubGraph1Flow -- 6 --> DoChoice2[Raman excluded] --> raw2scc{raw2scc}
     licel2scc{licel2scc}
     linkStyle 1 fill:none,stroke:red,stroke-width:2px;
     
  end

  subgraph "SCC Processing"
      web>WEB Interface] --> web_data(data submission) -- submit input file <br/> to the right SCC system# --> storage[(SCC_server)]
      storage[(SCC_server)] --> 
      code{Processing} --> 
      scc_products((SCC_Products))
      web --> web_sysconfig(SCC IPRAL Configuration <br/> 375=FR w/raman <br/> 376 FR w/o raman <br/> 377 NR w/raman <br/> 378 NR w/o raman)
  end

  subgraph "Data preparation"
      col[collect Raw Licel IPRAL binary Files] --> zipfile[1, 3 or more zip]
      zipfile -- extract RM files--> raw_prep((fa:fa-folder store files in <br/> 1 folder per hour from <br/> HH-30mn and HH+30mn))
  end
 
   subgraph "Process start"
     auto((Automatic <br/> processing)) -- date selection --> col
     user((fa:fa-hand-pointer-o User action))
     user -- date selection --> col
     linkStyle 1 fill:none,stroke:red,stroke-width:2px;
    end

   subgraph "BDD SIRTA or Other Data center"
     bdd[(BDD SIRTA)] --> raw_ipral((0a / ipral <br/> yyyy/mm/dd)) --> col
     bdd --> chm15k((CHM15K CBH)) --> raw2scc
     bdd --> ipral_config_files((Ipral Confog. Files)) --> raw2scc
     bdd --> temp((storage temp))
     raw_prep --> temp
     bdd --> scc_inputs((SCC input netcdf))
     raw2scc --> licel2scc_inputs((licel2scc reposetory <br/> Input filess)) -- Information of which SCC system# <br/> is linked to scc input file --> licel2scc --> scc_inputs
     bdd --> scc_outputs((SCC ouput netcdf))
     ipral_auto_config((IPRAL configuration <br/> in auto mode <br/> no radiosounding <br/> uses model)) --> |mode 0| licel2scc
   end
 
  subgraph "SCC API"
      scc_inputs --> |provide scc_input <br/> file & system number| scc_access{SCC_Access} --> web_data
      user --> web_sysconfig 
      scc_products --> scc_access --> scc_outputs
   end
 
classDef green fill:#9f6,stroke:#333,stroke-width:2px
classDef orange fill:#f96,stroke:#333,stroke-width:4px
classDef grey fill:#a98,stroke:#333,stroke-width:2px
classDef red fill:#f20,stroke:#333,stroke-width:2px
class user,auto green
class code,raw2scc,licel2scc,scc_access orange
class storage,bdd,store,climserv grey
class scc_outputs,raw_prep,scc_products red

%%{init: {'theme':'forest'}}%%
graph TB

 
subgraph "Legend"
  server[(processing server <br/> or data center)]
  interfaces>Interfaces]
  data((data set))
  user((actions))
  process_out((processing outputs))
  process_run{processing code}  
end

classDef green fill:#9f6,stroke:#333,stroke-width:2px
classDef orange fill:#f96,stroke:#333,stroke-width:4px
classDef grey fill:#a98,stroke:#333,stroke-width:2px
classDef red fill:#f20,stroke:#333,stroke-width:2px
class user green
class process_run orange
class server grey
class process_out red

Timeline
---

gantt
    dateFormat  YY-MM-DD
    axisFormat %d/%m
    excludes weekends
    title Automatic IPRAL Data Flow to SCC

    section Réunion
    Besoins exposés    :done, 2021-02-17, 1d

    section Raw2scc NRT
    Gathering&sorting raw data            :active, a1, 2021-02-18, 2d
    implement licel2scc     : a2, after a1, 3d
    section SCC Access API
    Implement new API on climserv      :active, 2021-02-18  , 3d
    Test data submission      :crit, a3, after a2 ,  2d
    
    section Raw2scc RA
    manual launcher    :after a3, 2d
    Test data submission with RS      :crit ,  2d

    section Retrievals coll.
    Get Products from SCC    : 1w
    Save Products in BDD or Loholt : 1w

## Comments

:::info
**Problèmes rencontrés ou questions?**

- Vérifier que la soumission à SCC avec le mode automatique code 0 pour utiliser le modèle demande ou non la pression et température au sol [name=CP]
- Réponse de CP: le mode automatique fonctionne bien avec le mode 0. CP a testé sur le jour 01/25/2021 il suffit de ne pas mettre de nom de fichier dans le champs 'Sounding_File_Name': "" , du fichier conf_ipral_375-376-377-378.py et de parametrer 'Molecular_Calc': 0, au lieu de 1
- Vérifier un cas de janvier 25 pour la soumission automatique et voir les fichiers reçus après traitement [name=MAD]
:::

###### tags: `SCC` `API` `IPRAL` `ACTRIS` `Lidar` `SIRTA`