## Climate and Forecast (CF) data handling in Python

### CF standards

- [CF Metadata Conventions](https://cfconventions.org)
- [UGRID Conventions](http://ugrid-conventions.github.io/ugrid-conventions/) (Unstructured meshes)
- [SGRID Conventions](http://sgrid.github.io/sgrid/) (Staggered Grid data model)

#### Other NetCDF-based conventions
- [Deltares NetCDF Wiki](https://publicwiki.deltares.nl/spaces/NETCDF/pages/41156617/netCDF)
- [BAWiki - NetCDF unstructured grid with subgrid](https://wiki.baw.de/en/index.php/NetCDF_unstructured_grid_with_subgrid)

### CF data libraries

- [Pangeo project](https://pangeo.io). A community platform for Big Data geoscience. Many of the following projects are somehow related to this community.
- [Technical Architecture](https://pangeo.io/architecture.html).
- [Project Pythia](https://projectpythia.org). Project Pythia has several resources for you to use to start learning how to use Python and the technology in the Python ecosystem for the geosciences.
- [Xarray](https://github.com/pydata/xarray)
- [related projects](https://docs.xarray.dev/en/latest/ecosystem.html#ecosystem).
- [UXarray](https://github.com/UXARRAY/uxarray). Extending Xarray to support unstructured grids (UGRID conventions).
- [xgcm](https://xgcm.readthedocs.io/en/latest/index.html): General Circulation Model Postprocessing with xarray.
- [Iris](https://scitools-iris.readthedocs.io/en/latest/)
- [Iris Mesh Data Model for unstructured grids](https://scitools-iris.readthedocs.io/en/latest/further_topics/ugrid/data_model.html) (UGRID conventions).
- [gridded](https://github.com/NOAA-ORR-ERD/gridded). A single API for accessing / working with gridded model results on multiple grid types. (It currently supports CF, UGRID and SGRID Conventions but it looks like it's going to be rewritten at some point on top of Xarray).

### Distributed/cloud/compressed data utilities

- [Dask](https://docs.dask.org/en/stable/) is a flexible library for parallel computing in Python.
- [Zarr](https://zarr.readthedocs.io/en/stable/index.html) is a file storage format for chunked, compressed, N-dimensional arrays based on an open-source specification.
- [Numcodecs](https://numcodecs.readthedocs.io/en/stable/) is a Python package providing buffer compression and transformation codecs for use in data storage and communication applications.
- [ArrayLake](https://docs.earthmover.io/) is a data lake platform for managing multidimensional arrays and metadata in the cloud.
- [Intake](https://intake.readthedocs.io/en/latest/) is a lightweight package for finding, investigating, loading and disseminating data.
- [Cubed](https://tom-e-white.com/cubed/index.html). Bounded-memory serverless distributed N-dimensional array processing.

### Data definition for I/O
- [Conduit - The Mesh Blueprint](https://llnl-conduit.readthedocs.io/en/latest/blueprint_mesh.html#)
- [Adios](https://adios2.readthedocs.io/en/v2.11.0/)

### Visualization
- [psy plot](https://github.com/psyplot/psyplot) used by C2SM libraries [icon-vis](https://github.com/C2SM/icon-vis) supports UGrid and ICON files
- [pygmt](https://www.pygmt.org/latest/index.html) includes projection functionality
- [Alpine ASCENT](https://ascent.readthedocs.io/en/latest/)

## Code sample for loading ICON NetCDF files with UXarray

Load an ICON NetCDF file using UXarray.

This proof-of-concept only works with the following UXarray branch: https://github.com/egparedes/uxarray/tree/gtdev
(required changes: https://github.com/egparedes/uxarray/commit/f8424c1d68d4f852f46e1d98460415ec892faaad)

grid_file = f"{global_grid_path}/icon_grid_0013_R02B04_G.nc"
grid_ds = xr.open_dataset(grid_file)

for var in grid_ds.values():
    if var.dtype == np.dtype(np.int32):
        var.attrs["start_index"] = 1

# TODO: investigate if this could be required in some cases
# for cf_role, var_name in mesh_topology_attrs.items():
#     if cf_role.endswith("_connectivity"):
#         grid_ds[var_name].attrs["cf_role"] = cf_role

grid_ds["mesh"] = xr.DataArray(
    -1, # A dummy value for creating the DataArray with the actual attributes
    attrs=dict(
        cf_role="mesh_topology",
        topology_dimension=2,
        node_coordinates="vlon vlat",
        edge_dimension="edge",
        edge_node_connectivity="edge_vertices",
        edge_coordinates="elon elat",
        face_dimension="cell",
        face_coordinates="clon clat",
        face_node_connectivity="vertex_of_cell",
        face_edge_connectivity="edge_of_cell",
        face_face_connectivity="neighbor_cell_index",
        edge_face_connectivity="adjacent_cell_of_edge",
        # boundary_node_connectivity="",
    ),
)
 
ugrid = ux.open_grid(grid_ds)
ugrid.validate()

ugrid.plot.nodes(title="Node Plot", size=1, height=500, width=1000)
ugrid.plot.edges(title="Edge Plot", height=500, width=1000)