# AZmet QA/QC

- Matt "plugged in" rounding to correct precision--only applied to *new* data (since the last month or so)
- Part of `azmetr` could be checking for correct precision
- Matt has a place online where we can edit the measured variables and it will propogate to derived variables (we think).
- Split tasks: modeling and workflow automation
- Phase one is alert Jeremy of extreme values and imputed values (a daily report table published with Quarto, for example)
- Could fit multivariate model as additional step (can't be raining all day and have high solar radiation)
- Could detect things in derived variables that we don't see in measured variables beause of transformations?

## Report refinements

Report currently uses **all data** for rule-based validations and **just one day** for forecast-based validations. This doesn't make sense. Need some flexibility in terms of what days are being viewed in the report and consistency between types of validations.

- [ ] Get forecast-based validation to work for past dates and potentially store results in a dataset somewhere

Then, there's two options:

1. Create reports with 1 week of data weekly and save them. Make a Quarto website to easily view weekly reports.
2. Make the report a Shiny app / flexdashboard with a date-range selector.

There's also the issue of the `pointblank` style tables not being very customizable and not being very readable sometim es (e.g. with segmented validations).

## Kickoff meeting (Oct 7)

Agenda:
- look at proposal
- set some dates

When CCT system was down Matt Harmon put some static QA/QC checks into place

Jeremey will send spreadsheet of active stations
Check if stations haven't reported for > 6 hrs

Dynamic QA/QC
- Probability distributions for paramaters using historical data, flag if outside some threshold
- Look how other systems do weather QA/QC
- https://data.ess-dive.lbl.gov/view/doi:10.15485/1823516
- https://github.com/WSWUP/pyWeatherQAQC
- Compare to climatology products (e.g. Daymet) or use those product for modeling
- Compare to forecast data products (e.g. from NOAA). Use ensemble forecasts to create interval for QA/QC.

## Initial Meeting
AZMet update
- equipment maintainence / optimization
- modernization of data workflow
- need to update website

QA/QC needs
- sensor ranges
- check for physically impossible values
- Some (but not all) stations have data loggers that deal with small errors (e.g. RH = 100.1%)
- Collection frequency = hourly
- Maybe a Shiny dashboard (or other data visualization) to see the data plotted in addition to just alerts
- Basic Shiny apps for this exist
- Done some resarch on industry standards on QA/QC for environmental sensors. Can use this as source for QA/QC checks
- Reach goal: inter-station comparisons

Q: Is there software already in existance for meterology QA/QC?
A: Haven't seen anything from mesonets. Need to look around.

- FluxNet has shared tools for QA/QC

David suggestion---for every data point, what's the probability that it's an error given history? What's the probability it's a "true" data point?

Matt Harmon
- was responsible for porting AZMet to more modern infrastructure
- dataloggers -> JSON -> python scripts
- some basic range checking and normalization is in python object
- Future idea that ranges would get pulled from a database table
- E.g. some of the weather stations are in golf courses that use sprinklers
- ranges in database table could be dynamic
- Simple validations built into code that pulls data (e.g. laws of physics) and have more sophisticated validations that rely on spatial and temporal variation in another layer.

3 layers:
1. laws of physics checks---hard coded
2. range checks from ranges in database (seasonally dynamic?)
3. spatio-temporal checks on data output including calculated values

Layer 1 in existing python code (or can be added)
Layer 2 maybe added to existing python code
Layer 3 dashboard

Notifications:
- Some of this done currently using SQL database
- Whatever tools do QA/QC just need to be able to interact with SQL database or REST API for notifications

Q: Are all tables exposed as REST API?
A: Depends on what we want to do with data. Currently API hides some things---can't address a particular table or see metadata without authorization.

Data Integrity:
- Previously some manual manipulation of data coming off data loggers without any tracking
- Now manual modifications and updates are versioned

Q: Do you want provisional data (ASAP) and validated data? What's the desired turn-around for validated data?
A: Not a problem to have near-real-time data being provisional. Don't know what the turn-around should be for validated data. Need to figure it out.

Q: Could you define what kind of dashboard or automated reporting would be most useful?
A: Need to think through it still

- Need a separate dashboard for QA/QC for use in the field for diagnosing problems (doesn't need to be constrained to AZmet website)
- Separate dashboard for data consumers
- For website, focus on tabular data presentation (`gt` package?)
- For website, haven't gone beyond built-in capabilities of drupal
- Haven't gone into visualizations for website

API conversation
- Whenever we (David and Eric) only have read access to database that's a good thing.

Next steps:
- Drafting proposal for data incubator
- Jeremy will get it started and send around

# meeting

Loading time isn't a big issue since only used internally

Likes the shiny app interface with date selector

AZMET in good position to fund continued work

First step:
Consistency checks, date slider, daily data shiny app

Report just pulls data from azmetr, doesn't need targets pipeline