CR CARS-ARES Webinar: QA tests et python tool presentation

CR CARS-ARES Webinar: QA tests et python tool presentation === ###### tags: `CR` `CARS` `SCC` `ACTRIS-EU` `Earlinet` :::info - **Location:** Visio - **Date:** May 18, 2021 9:00 AM (CET) - **Agenda:** 1. QA tests explained and showcases - [name=Volker Freudenthaler] 2. Software tool for QA tests - [name=Livio Belegante & Victor Nicolae] - **Participants:** - [x] Chris Pietras (CP) - [x] Doina Nicolae (DN) - [x] Volker Freudenthaler(VF) - [x] Livio Bellegante (LB) - [x] Victor Nicolae (VN) - [x] earlinet <earlinet@actris.imaa.cnr.it> - **Contact:** Doina Nicolae <nnicol@inoe.ro> - **Host:** DN - **Reference:** - [CARS-ARES Webinar](https://hackmd.io/@scc-algo/SkD0WWdbO) ::: :question: SIR Station preparation --- - ==CP:== Distribution of the tool => Pypi? - ==CP:== How to deal with raw licel data to use the tool? :lantern: Slides of the webinar --- - [Slides de Volker Freudenthaller QA-test-measurements requirements](https://drive.google.com/file/d/1mg0GM7DQVi3ilj5tdNr8GfflZWN1Aty3/view?usp=sharing) - [Slides de LB & VN - Software tool for QA tests](https://drive.google.com/file/d/1HKwl8Lhs08U-8ATppk30duupIpcF9GXq/view?usp=sharing) :closed_book: Tasks that SIR Station will have to do after the CARS meeting -- > IPRAL Team [name=MAD, Marc Antoine Drouin][name=CP, Chris Pietras][name=FL, Florian Lapouge][name=PD, Patricia Delville][name=CB, Christophe Boitel][name=JFB, Jean François Ribaud] > SCC Team [name=GDA, Giuseppe D'Amico][name=LM, Lucia Mona][name=IM< Ina Mattis] > CARS QA Team [name=VF, Volker Freudenthaler][name=LB, Livio Bellegante][name=VN, Victor Nicolae] ### IPRAL Team: - [ ] ==CP, FL, PD== Faire des mesures de QA dès que possible de bones qualité. DOnc plutôt tôt le matin ou le soir pour les telecovers et la depolar. tester des mesures sur 30 minutes pour améliorer le moyennage - [ ] ==CP== Refaire des mesures de dark moyennage 5mn ttes les 30 minutes pour minimiser le bruit et voir - [ ] ==CP, MAD== Voir à récupérer l'outil de traitement des QA par CARS et comment l'installer sur Climserv ### SCC Team: - [ ] Le groupe SCC doit travailler avec le groupe QA pour voir comment implémenter l'outil de traitement des QA dans SCC en s'appuyant sur l'existant ## Notes  - Introduction from DN - Discussion about new web site for cars with presentation - Description of QA tests by Volker - list of QA: trigger delay, dark, RF, Telecover, depolar: see which needs analog+photon or just analog - Trigger delay: important how to make correct measurement and check if system allow correctly dark the system with the hole through the cover where the fiber is inserted Volker show the distorsion in signal pretrigger and how to correct it - Dark signal substraction. We made that already @1064 for IPRAL with Juan and it was Ok for Volker. We do not do it anymore from now. Check if dark signal is stable over minutes or hours. Make measurements over 5 min average every 30 mn. Do it again a couple of times. - RF @1064 exemple: shows that we need at least 2kms of free aerosols. - discuss also to check radiosounding quality also - please make 30 min long measurements & cut signals at 30kms - also put all channels in one file - Depolar: - make also longer measurements for good average measurements - if too much noise, volker has problems to see where is clean air and determine depolaization bias and uncertainties of the bias calculation will go down and is much better - Volker shows the error of the depolar and shows what was calculated for IPRAL :+1: see slides - Telecover: shows change of atmosphere during telecover. - shows several cases, some good and some not good enough - some are too noisy because not averaged enough. Also please make measurements at night. - Check good wheather conditions to make sure you will have good QA analysis that has consequences like minimum systematic error in further data analysis. - Software tool for QA analysis: Livio - python tool for each QA tests - uses raw licel files but see lokks like it should be averaged before. The tool reads only one file per analysis ---> see with MAD and CB if it is easy to average several binary files and keep the same headers and format. - CP ask how the tool will be shared. Pypi... other... - Exchange with Ina and all how it would be better to rely on SCC since most of the features are there averaging, range corrected etc... and so Livio and Victor could contribute in SCC development to add option to analyze QA files. CP point it out that we need external tool since Volker needs also specific ascii formated files to perform the QA deep analysis. - However Livio & Victo will work with SCC developer to see how to integrate in a next version of SCC. - CP mention the pb of averaging binary files - Victor shows exemple of analysis. how to configure different things for systems - discuss if the tool is also ok for coaxiallidar system: yes - Volker mention if tool compatible with East2 or West2 instead of north2 since some system uses that because noisy signal if north sector is obstructed somewhat and signal is too low: Livio reply it could be possible - CP mention the averaging problem of raw licel files. We will see waht coud be done. - Volker will open a forum question of the tool some everyone could contribut and ask what is necessary in this tool. - CP a rempli le topic dans le forum avec les recommendations: *Thanks Volker for opening this topic in the forum, Today at the end of the CARS meeting, Doinae requested to the Pis to share their recommendations to help Livio and Victor to complete the python tool that provide first evaluation feedback of the QA test before sending them to Volker or to SCC. So, below are recommendations that I shared during the meeting: 1- We could test this first version of the tool with our data to make sure our Licel raw files are suitable. Because if I remember correctly, Ioannis for licel2scc had to tune a bit the reader to make sure it works for all stations (I guess for IPRAL we have different header that needs to be taken into account). We could also verify that the output files are suitable for Volker deeper analysis. FOr exemple, Volker asked if possible we could limit the QA range files to 30km to speed the processing. For IPRAL we acquire up to 60km. So, if the tool is about to be upgraded, it could then limit the output QA data to 30kms. 2- Also when we run our lidar we always make same raw files (except name) so to speak, 900 averaged profiles in all channels whether raman are activated or not. So it is easier to handle. For QA (telecov+depol) we can make longer acquisition but we will get more raw licel files. So we need to concatenate those binary files one way to another for the python QA tool to be able to analyse them. Or upgrade the tool. 3- Once the PI's have firs tested the tool, it would be great to easily benefit of the upgrade when Livio and Victor completed them. It would be great if the tool could be shared through Pypi or equivalent. 4- Ina mentioned that many calculation are already implemented in SCC that could be re-used for QA analysis using pre-processed files. This is true that there are advantages for PIs since they could just submit the files to SCC as usual using licel2scc converter and then implement in SCC a specific routine that takes into account telecover sequence. I guess depol is already there since we use already licel2sccdepol. However, i am not sure, but it might take time before such QA analysis could be implemented in SCC ? especially to make sure that we could get quickly suitable files to provide to Volker. In the meantime we could greatly appreciate the current python tool in order to perform our next QA tests and quickly analyze them and provide them to Volker.*