---
# System prepended metadata

title: 2023 DMAC

---

# 2023 DMAC

## IOOS Office Updates - Becky Baltes

- Becky is the acting office chief while Derick is distracted
- NOAA strategic goals
    - Climate ready nation
    - Equity
    - Accelerate growth in an information based blue ecconomy
- IRA planning
- 2024 Code Sprint in DC
- Coastal and Ocean Modeling
- Highlights
    - Gliders
        - github.com/ioos/glider-dac
    - Marine Life
        - Passive Acoustics SoundCOOP
    - HF-Radar
    - HAB
        - Working through Congress and likely to get their blessing
- Product updates
    - ioos.us
        - New tiling tech
    - Environmental sensor map
        - new version coming out shortly
    - Model viewer
        - Also a new version coming soon
    - Metrics
        - Built from the region progress reports

## Global Ocean Monitoring and Observing (GOMO) and IOOS: Future opportunities for aligning data communities of practice (Ann Zinkann, Alyse Larkin, Cindy Garcia/GOMO)

- What is GOMO
    - Global Ocean Monitoring and Observing program
    - Provide and support high quality global ocean observations...
- Support ~3k platforms
- No formal data strategy
- What now
    - Develop a streamline, robust, and integrated the GOMO data ecosystem
- Alignment with IOOS Data Management

## CIOOS Updates (Ray Brunsting/CIOOS & Scott Bruce)

- Update number 5, 5 years in
- About 2% of the funding as IOOS
    - Not really paying for gear to be put in the water
    - Focused on the data side
- Working together alot
    - National committees
    - Shared tooling
    - Shared infra
- 2 year funding cycle
- CIOOS has become a regional association of GOOS as an IOOS sibling rather than combing into a North America region
- Infra
    - Each region runs their own CKAN and ERDDAP
    - Institutions often run their own ERDDAP and/or CKANs
    - National CKAN that harvests from the regions
- National tools
    - Data explorer
        - More comprehensive tool, filter, aggregate, and download various datasets
        - Can fall back to aggregate ERDDAP data if CKAN isn't avaliable
            - Integrated GLOS data this way
    - Catalogue map
    - Data catalogue
        - Driven by CKAN
            - Hassle Matt Foster for more info
        - variable categories
            - Bioecosystems
            - biogeochemical
            - cross-displinary
            - Physical
        - Integrated into Ocean InfoHub
        - Choosing to federate early has worked out well
- Atlantic
    - Hurricane Dashboard
        - Historial storm details
        - nea-realtime data nad models predictions from Evironment Canada
        - Under development, talk to Scott
- Saint Lawrence
    - Navigation
        - Boating aid too
        - Current, wind, tide, weather forecasts
    - HAB monitoring
        - Model developed by Department of Fisheries and Oceans
- Pacific
    - Ocean Connect - https://oceanconnect.ca/
        - Salish sea focused
        - Trying to use higher res model compared to what Windy is doing
        - Bringing in 
- Areas of focus
    - Continued development of national and regional tools
    - Integration with global data catalogs
    - Model data integration
    - AI

## OTN, ATN, Nodes, and OBIS (Jon Pye/OTN, Angela Dini (OTN))

- History
    - Started in 2008
    - Anomaly for Canadian programs as its supposed to work with foreign partners
    - Initally had a single centralized station with 300 million detections
    - Infrastructure has moved funding sources to a 6 year repeating cycle
    - Now coordinating other communities and helping them stay compatible
- Nodes
    - 'Everything is related to everything eelse. But near things are more related than distant things'
    - More trusting of neighbors with their data
    - PostGIS database
    - But it more a social problem than tech problem
    - OTN can host nodes if folks don't have the funding to do so locally
    - Hosted by others
        - SECORA
            - Hundreds of millions of detections
            - Hosted by Axiom
        - MARACOOS and Smithsonian
            - ...
- US Nodes
- Publication pipelines
    - OBIS
        - Any OTN node can push up to OBIS
        - Python based, any node manager can run
        - 
- Data sharing inititatives
    - DaViT
        - visualizes acoustically detected animal movement data on a per speiecs and per project level
        - Adding more QC checks
    - Rshiny
        - subsetting and downloading from PATH
- Animal tracking network
    - AniBOS, wifi equipped seals

##  Towards a Prototype National Harmful Algal Bloom DAC: Successes, Challenges, and Lessons Learned (Rob Bochenek/Axiom)

- Establish a California regional HAB hub
- Imaging Flow CytoBot
    - real time photos of critters
    - very ad-hoc research data processing currently
- Transitioning to operations
    - ML models from random forests to convolutional neural netowkrs
    - oerational data processing pipeline for processing and classification
    - improve quality of traing data set through data governance
- Impacts
    - improved accuracy from 70% to 90%
    - moved latency from weeks to near real time
- Regional expansion
    - National implementaton plan to be released soon
    - Work with regional pilots to assimilate models
- Processing pipeline
    - Rsyncing raw data
    - Instruments need to push data due to federal IT reqirements
    - Dask for analysis
    - ML model classifiers are run in real time on a 4 V100 GPU cluster
        - 12-15$ an hour to get that power in the cloud
        - machine in Axioms data center
        - PyTorch and Tensorflow models run on the same feed for comparison
    - WHOI dashboard for instrument operators
    - Postgres and Streamlit for alert and other mangement for operators
    - Lots of the processing was managed by people
- Focus on performance
    - calculations are done as close to data as possible
    - FastAPI
    - reduced query times from 20+ seconds to <1
    - Pre-calculating winners
- Assing data using streamlit
    - can be worked on by many levels of engineer/scientists
- Visions
    - Want to make it plug and play for IFCBs
        - Rapidly view sample images on dashboard
    - Compare regional classifiers side by side
        - Also how much processing can happen on the instruments?
    - Annotated image libraries
    - Automated product development
    - Script libraries for data archiving
    - Regional Diversity
    - Management Products People Want

##  SoundCoop - Passive Acoustic Monitoring Access Network (Carrie Wall/NCEI)

- Federal acousting monitoring and management
- Existing portal capacity to compare or access standard projects
    - SanctSound
    - PACM
        - Passive acoustic cetacian map
        - Whale detection data over time
    - NCEI
        - Passive acoustic data viewer
- How to leverage all the systems by advancing standards and community tools
- Passive Acoustic Monitoring National Cyberinfrastructure Center (SoundCoop)
    - Community focued
    - IOOS, BOEM, NAVFAC, ONR
- Progression - 3 year effort
    - 1 - Build national repository
        - NCEI archive
    - 2 - Integrated access
        - Build infrastructure to discover and access existing raw files and data products across seperate repositories
    - 3 - Visualization tools
    - 4 - ?
- User profiles
    - Temporal sound level analysis in the arctic oceans
    - Spatial sound level analysis of NERACOOS, CeNCOOS, and SECOORA
        - Monhegan and Petit Manan
    - Offshore energy development datasets
        - Integrate BOEM and state funding
        - North atlantic right whale detections
    - International datasets
        - Synergy with international effors
        - NEFS - Tim Rowell
        - Woring with AWI/OPUS and JOMOPANS
- Tasks
    - Run MANTA to create sound level metrics
        - daily files of 1 min resolution hybrid milidecade band spectra
        - Standalone matlab, no license needed
    - Evaluate pypam
        - As comparitive tool for sould level metrics
    - Archive data and products at NCEI with a NetCDF standard
    - Establish workflows to access data in cloud 
    - ...
- Passive acoustics community is largely producing CSVs
    - No idea what NetCDF is or why metadata is helpful
- Power of millidecade NetCDFs
    - Audio data
        - 4 TB, 51k files
        - ~ 77 MB, Wav
    - Hybrid Millidecade spectra
        - 5.4 GB
        - 1,440 files
        - 15 MB
        - CSV, netCDF, and pngs
    - Hybrid Millidecade spectra - daily
        - ...
- Cyberinfra framework
    - 3 PAM repos
        - NCEI Google Cloud
            - Hybrid millidecade from 14 recording sites across 8 sites using MANTA netCDF
            - https://www.ncei.noaa.gov/products/passive-acoustic-data
        - MBARI AWS
            - Hybrid from 1 site using pypam netCDF
        - Axiom in house
            - Hybrid millidecade from 1 site using MANTA
    - Audio files
    - Standard derived products and metadata
    - Common software stack
        - Jupyter
        - MANTA
        - ERDDAP
- Visualizing
    - SoundCOOP portal
    - points show recording sites
    - Level one product
- Development of community tools
    - Notebooks
        - How to access netCDF
        - Everyone currently has to recreate the wheel
        - Comparison with other types of data
- Next steps
    - Finalize processing of all datasets
    - Develop user environment to pull raw data from repository and create standard metrics
    - Finalize portal and co-visualize results
- Working on making sure it's possibly to go back to the raw files from the daily millidecimated netCDFs

## Data Management Workflow and Challenges in Developing a California Ocean Acidification and Hypoxia Data Portal (Marine Lebrec/CeNCOOS)

- Objectives
    - improve data quality, interoperability, and access
    - streamline data ingestion for enhanced
- Ingesting new datasets
    - NOAA PMEL West Coast ocean acidification crusies
        - via NCEI
        - CTD
        - Underway PCO2
        - Petropod abundance/disolution
        - Using the Axiom research workspace
    - Co-located environment and plankton monitoring
        - via ERDDAP
    - MBARI underway PCO2 via MySQL
        - 1993-present
        - cruise data aggregated via 1km grids
- Water quality dashboard for shelfish growers
    - Update hourly
    - 14-day trend
    - QARTOD filtering
    - Growers wanted a simple user interface to access water quality data
    - JSON parameter file for each shore station
    - build URLS for ERDDAP query
    - Fetch data via erddapy, write to pandas df
    - Calculate hourly mean, remove failed QARTOD
- Future direction
    - Integration of other datasets
    - Development of next gen dashboards with Axiom
        - Shelfish growers
        - Water management
        - Fisheries
        - MPAs
    - Notification/alert system for users
        - Still scoping

##  Standardizing Marine Biological Data working group update (Tim Van Der Stap/Hakai)

- Started during the 2019 code sprint
- Aligning marine biological data to Darwin Core to share to OBIS
- Practical and informal
- Distracted by the idea of lunch

## ATN data pipeline updates - GTS/OceanOPS and NCEI (satellite telemetry sharing) (Megan McKinzie/ATN)

- Mutli agency funding so serving their needs as well as the scientific community
- DAC
    - run by Axiom
    - Recieves data from multiple sources/tag types
    - Enables archiving at NCEI,...
    - Primarily satelite data from ARgos
- Serving animal borne profiles to GTS
    - Proposed
        - PI, taxon, or region specific configs
        - Additional tests
- Animal borne profiles
    - increase in ocean model quality and decreased error as well as storm and weather predictions
- Data processing pipeline
    - Metadata in ADR
    - Near real time auto ingesting
    - Every two hours
    - Run through QARTOD
- viz.pmel.noaa.gov/omsc/
- AniBOS OceanOPS dashboard
- Need to translate QARTOD flags into GTS flags
- ATN Seal of Approval

##  Application of Cloud-Native Solutions to challenging datasets (Kelly Knee/Audra Luscher)

- Reaching for the cloud
    - Decrease barriers for entry
    - Prototype roadmap
        - Direct access from S3
        - Ingest
        - Storage and discovery
        - Processing
        - Analysis and Presentation
    - Case studies
        - NCDIS 40 year reanalysis - CO-OPS
            - Outlooks for sea level rise
            - Climatological
            - Gridded 
            - National Coastal Data Informations System Multi-daecadal Year Reanalysis
            - Long term reanalysys of water levels and waves
            - Integrating models and observations to predict flooding between tide stations
            - Fills extensive gaps between observation locations
            - 500m grid
                - ~70 TB of storage
            - Monthly high tide outlook
                - Sea level trends
        - National water model
            - 5 GB per day, updated hourly
            - networked
            - 2.7 million reaches
            - Additionaly hydrologic information on 1km, 250m, and 100m grids
            - NODD S3 bucket
                - Lat/lon not within the existing files
            - Kerchunk to add lat long

## NCOS Coastal Reports and Ocean Finder: Real Time Geoanalytics of IOOS Data Streams in Support of Marine Spatial Planning (Rob Bochenek/Axiom)

- If you end up in Anchorage, please drop by Axiom to visit or if you need a place to work
- Coastal Reports
    - Building on Ocean Reports (previous tool)
    - Draw a polygon and produce analytics and reports
- Ocean Finder
    - Spatial suitability analytics across user defined criteria across multidisciplinary datasets across the US EEZ
    - Jupyter lab, Spark, Sedona, FastAPI, Parquet, GeoTIFF, netCDF
- H3 heiarchial, spatial indexing
    - triangles had 12 neighbors
    - squares have 8 neighbors
    - hexagons always have 6 neighbors and share edges and vertices
    - Aperture 7 mesh

## Extending data services beyond the US EEZ (Felimon Gayanilo, GCOOS)

- Hard to know what is going on in the Gulf of Mexico without knowing what is going on around the Yucatan Peninsula

## GLOS Updates: Seagull Update/Overview + Technical Overview (Tim Kerns, Joe Smith/GLOS)

- Wanted to build a tech platform

## HF Radar Range Series Archival Project Status Update (Shane St. Savage/Axiom)

- Goals
    - Organize near-real time HF radar range series and instrument config files from all IOOS HF operators
- New features
    - Data and config download via ERDDAP
        - Broke tables and had to switch to the lazy loading of directory only access
- Next steps
    - Expand to more operators
    - Enable multi-file downloads with a UI tool to generate curl commands
    - Improve UI plot rendering perfomance
    - Make deployment more robust

##  Serving QC solutions: New approaches (Eugene Burger/PMEL)

-  https://github.com/NOAA-PMEL/QCaaS

##  Water Level QC AI (Lindsay Abrams/CO-OPS, James Spore, Hassan Moustahfid/IOOS)

- Current QC workflow
    - Primary and backup sensors
    - Onsite QC
    - Preliminary
    - Manual processing and verification
    - Takes a lot of people and resources
- Common issues
    - Spikes
    - Missing
    - Flats
    - Rare events
        - but actually legit, so should be identified as good
        - Tsunami
- Ideal AI workflow
    - preprossing
    - classify good and bad using AI model
        - currently here
    - correct bad data automatically using secondary algorith
- Model testings
    - Logistic regression
    - Random Forest
    - Gradient Boost
    - Neural Net
    - All had around 99.7% accuracy, except regression was around 98.9%
- Neural network approach
    - Inputs
        - primary
        - backup
        - predictions
        - residual level
        - sigma water level...
    - two hidden layers
    - scikit learn, TensorFlow, Keras
- Regional model training
    - Started building models for specific regions for better accuracy
    - NorthEast tends to be more accurate than all but Hawaii
    - NW did especially poorly
- NE Model
    - loss stabilized
- NW model
    - loss never stabilized
    - Some bad backup data (Tacoma) caused some overfitting
- Takeaways
    - ML approaches can be used to classify good/bad water level data
    - Neural net model is best approach from testing so far
    - Quality of data used for training is very important
    - GPUs can be used to preform many experiments at once and accelerate AI/ML research
    - An automated AI-enabled QC application would reduce CO-OPS resouce requirements and could be used as a community tool
- Next steps
    - Troubleshoot issues with AI model for classifying bad data
    - Bad data is rare
        - Make sure it's better represented in training datasets
    - Develop algorithms to correct bad data
        - Initial results are promising using a regression model
- Using HPC
    - Converted to Parquet to work with RAPIDS
    - OpenACC Hack-a-thon allowed use of GPUs and HPC
- Questions for crowd
    - Employed automatic QC?
    - Interested in a QC tool that automatically identifies and corrects bad data
    - What gap filling algorithms do you currently use?
        - Recomendations on AI/ML methods for gap filling
    - Opportunities to collaborate
        - Can we use your water level datasets to test our model

##  IOOS/NOAA Cloud Sandbox Update (Patrick Tripp/RPS, Tiany C. Vance/IOOS)

- Running regional coastal models in the cloud
- Gulf of Maine is an OFS domain
- Lynker is currently deploying the sandbox on NOAA's cloud

##  IOOS Sensor Map ERDDAP Integration Status Update (Shane St. Savage/Axiom)

- RA should have more control what shows in the map
- Beta
    - https://beta.sensors.ioos.us/
    - Limited to data served by RAs
        - may include other sources to augument spatial coverage
    - Discovered in IOOS catalog
    - Served via ERDDAP tabledap
    - Queries IOOS Ckan search daily
    - Harvests data every 15 minutes
    - Current status
        - Monthly processing reports emailed to RA data managers
        - ~12k stations where old one had ~46k
        - Transitioning back to using the IOOS catalog for most RAs
- Stable
    - Kafka > TimescaleDB
    - Hex binning
    - Temporal binning in API endpoints
    - Comprehensive view of data but doesn't reflect RA curation
- Next steps
    - Quality of life tooling for ERDDAP
        - docker-erddap
        - erddap-gold-standard
    - Reaching out to RAs to sort out ERDDAP service avaliablity or data issues
    - Improve operational status reporting
    - Coverage gap analysis

## ERDDAP update

- New NDBC division chief
    - Meet the regions where they are
    - IT systems are frankenstein monsters
        - Need to get that fixed
- Priorities for the next few years for ERDDAP development
    - Improve the testing workflow
        - Takes maybe a day to run
    - Sharing the private backlog of issues and request publicly
    - Wants to move to more standard libraries but need to be aware of memory usage since it's already very heavy
        - Reduce the complexity of the current codebase
    - Part of the role of the committee
    - Update directory structure to match other maven projects
- ERDDAP for archiving
    - ERDDAP slaps a timestamp in NetCDFs on build
        - in the history
        - doesn't matter if the source data hasn't changed
    - RAs are supposed to generate netCDFs manually in the WAF
- What does CIOOS need out of ERDDAP/what are their plans?
    - Friendlier UI and downloader



## Links

- Quality Control as a service - https://github.com/NOAA-PMEL/QCaaS
- ERDDAP util Tomcat log parsing - https://erddaputil.readthedocs.io/en/latest/tomtail.html
- Axiom intake repos - https://github.com/search?q=org%3Aaxiom-data-science+intake&type=repositories
- Intake ERDDAP - https://github.com/axiom-data-science/intake-erddap
- Xpublish Intake serving - https://github.com/axiom-data-science/xpublish-intake
- Intake Axiom - https://github.com/axiom-data-science/intake-axds
- Intake CO-OPS - https://github.com/axiom-data-science/intake-coops
- ERDDAP getting started - https://coastwatch.pfeg.noaa.gov/erddapinfo/index.html
- Axiom hacking - https://github.com/axiom-data-science/hackathon-tabular
- Axiom intake catalogs to load model data - https://github.com/axiom-data-science/mc-goods
- Coastwatch satellite course? - https://coastwatch.pfeg.noaa.gov/courses/satellite_course_info.html
- NCEI passive acoustic data - https://www.ncei.noaa.gov/products/passive-acoustic-data
- 