# Data Tidying

Xarray provides flexible structures with which to store and access data. However, appropriately coercing data into a format that simplifies subsequent processing and analysis is often a difficult first step for users of Xarray.

This page discusses common data 'tidying' steps, and presents principles to keep in mind when organizing data in Xarray. We also point out helpful extensions to simplify and automate this process for specific dataset types like satellite imagery.

### Tidy Data

Hadley Wickham developed the tidy data concept for tabular datasets in the R programming language ([paper](https://vita.had.co.nz/papers/tidy-data.pdf)). It presents three main principles:
1. Every column is a variable
2. Every row is an observation
3. Every cell is a single value

You can find more in-depth explanations of tidy data and the body of work it has inspired [here](https://r4ds.hadley.nz/data-tidy.html).

Here, we articulate 'tidy' principles for n-dimensional gridded data as represented by an Xarray Dataset.

### Thinking of your data as a data cube

Data is stored and made available to users in a variety of format and structures. Commonly the first step of many scientific workflows will be to assemble a data cube of the form `(x,y,time)` out of a collection of files representing spatial and temporal subsets of data. There are different ways to stitch together many files into a cohesive data object and the structure of the stitched together object will either make subsequent analysis smoother or more difficult. Tidy data is about structuring data to simplify subsequent analysis.

## Understanding & using Xarray structures

Xarray is built around three main structures: `xr.Datasets`, `xr.DataArrays` and [DataTrees](https://xarray-datatree.readthedocs.io/en/latest/data-structures.html). Think of `xr.DataArrays` as single variable objects that contain dimensions and coordinates. `xrDatasets` contain multiple data variables. Each data variable of a `xr.Dataset` is a `xr.DataArray`. DataTree objects are structures used for organizing workflows with multiple, un-alignable `xr.Datasets`. Returning to the concept of a data cube, `xr.DataArrays` are cubes, `xr.Datasets` are a collection of cubes that share a common “domain”, and DataTrees are collections of cubes, that need not share “domains”. For simplicity, this page will focus mostly on `xr.Dataset` objects.

HTML repr of a standard `xr.Dataset` object.

Before discussing principles of tidy objects in Xarray, we provide non-technical definitions of Xarray objects that relate to how they can be used when organizing data. For more, see [the Xarray reference documentation](https://docs.xarray.dev/en/stable/user-guide/terminology.html).

| Term | Definition |
| ------------------------- |:------------------------------------------------------------------------------------ |
| Data Variables | Physical observables that are the subject of the dataset. |
| Dimension Coordinates | Variables defining the domain of observable(s). |
| Non-dimension coordinates | Metadata that is best expressed as arrays sharing dimensions with the data variable. |
| Attributes | Metadata that is static, commonly key-value pairs. |

## Tidy Xarray

## Principles of Tidy Gridded Data

### 1. [Dimensional coordinate variables](https://docs.xarray.dev/en/stable/user-guide/terminology.html#term-Dimension-coordinate)
Minimize # of dimensional coords; only what is necessary to describe shape of your data
### 2. [Non-dimensional coordinate variables](https://docs.xarray.dev/en/stable/user-guide/terminology.html#term-Non-dimension-coordinate)
Non-dimensional coordinates can be numerous. Each should exist along one or multiple dimensions
### 3. [Data variables](https://docs.xarray.dev/en/stable/user-guide/terminology.html#term-Variable)
These should be observables rather than contextual, each should exist along one or multiple dimensions
### 4. Contextual information (metadata)
Metadata should only be stored as an attribute if it is static along the dimensions to which it is applied
If metadata is dynamic, store as coordinate variable
Metadata attrs should be added such that dataset is self-describing (following CF-conventions)
### 5. Variable, attribute naming
Where possible, use cf-conventions for naming
Variable names should be descriptive
Variable names should not contain information that belongs in a dimension or coordinate. (ie. information stored in variable name should be reduced only to observable)
### 6. Make use of, work within the frameworks of other tools
Tools like STAC, open data cube, cf.xarray, pystac, stackstac [and many more] maker tidying possible (+ smoother), especially with large datasets

Diagram of Xarray object components:


---

### Guiding questions for tidying data[^1]

To help guide categorizing the data and metadata at hand accoring to Table XX, we suggest considering the following questions:
1. What type of questions do I want to ask of this dataset, what kind of structure will facilitate that analysis?
2. What is the meaning of the data, and how can its *structure* best reflect its *meaning*?

## Roadmap of common data tidying steps

This section discusses common 'data tidying' steps in a typical scientific workflow.

### Before reading data into Xarray

Organizing and storing large numbers of files covering varying spatial and temporal domains so that they can be read into Xarray efficiently and intuitively can be a non-trivial task in itself. This one step where descriptive and conventional metadata can be very beneficial. It is worth spending some time understanding the layout of the dataset, what data you need to answer your scientific questions, and the overall data structure that would help you to ask and answer those questions. Then, you can consider which tools might be appropriate to help read the data into Xarray. Some options to consider are [stackstac](https://stackstac.readthedocs.io/en/latest/), [odc-stac](https://github.com/opendatacube/odc-stac), [PySTAC](https://pystac.readthedocs.io/en/stable/), [Intake](https://github.com/intake/intake) and [Kerchunk](https://fsspec.github.io/kerchunk/).
### Reading data into Xarray

Data is often distributed as individual files that need to be combined in to a cube to answer the questions at hand. Often, these files are separated into spatial and temporal subsets and the first task of the user will be to read in the desired data variables, and temporal and spatial regions of interest, and construct a tidy data cube with which to pursue further analysis.

`xarray.open_mfdataset()` is a very useful function at this step. If your dataset follows STAC metadata specifications, tools like stackstac and odc-stac can be helpful for querying and subsetting large catalogues of data.

### Assembling and tidying a data cube

After the relevant files have been accessed, the next step is typically to assemble different objects into one cohesive and self-describing data cube. Usually, this will be an `xarray.DataArray` or `xarray.Dataset` with the form `(space,time)`. This step often entails:
1. Parsing filenames to access metadata
2. Adding new dimensions,
3. Converting metadata in individual files to non-dimension coordinate variables,
4. Renaming dimension names,
5. Concatenating different objects along a dimension,
6. Merging cubes to form a dataset,
7. Adding additional metadata based on dataset reference documentation, or to follow metadata conventions (e.g. [CF](https://cfconventions.org/), [STAC](https://stacspec.org/en))

For larger-than-memory data when dask arrays are used, rechunking usually occurs at this step.

### Helpful Resources
[Project Pythia tutorial on CF/netCDF](https://foundations.projectpythia.org/core/data-formats/netcdf-cf.html)

### Data cube --> Analysis ready

> [name=Deepak Cherian] I added text above that ended up merging this section and the previous one. Let's condense?
> I think the specific examples are best added under subsections that share titles with the list above. Though if there isn't much text, maybe we can just add more text under each list element. I believe latex calls these "description" lists.

After step 2, the data cube often appears 'tidied', however frequently there are additional manipulations that must be performed to truly make the data ready to work with (or, *analysis-ready*). These steps can vary widely and be specific to individual datasets but are very important to creating objects that will simplify future analysis. Examples of this can be found (here - link to itslive example where there are many many data variables, most should be coordinate variables) and here (link to HLS example with bit packed masks). Any others?

## Common user-hangups

#### 1. Embedding a dimension in a variable name
Frequently-observed example: Data variables such as `air_temp_t1` and `air_temp_t2` to represent data about an observable (`air temp`) collected at 2 different points in time (`t1`, `t2`). (insert image/code of corresponding dataset).

This is an example of the data structure not matching its meaning. A helpful way to catch these instances is to check if you are trying to provide a description beyond the direct observable in the variable name. In this case, the variable names are being used to describe the observable, but also where it lies along a time dimension. Not only is this clunky, but it prevents you from using built-in xarray functionality like grouping over dimensions (making subsequent organization and analysis much more difficult) insert code example

#### 2. Metadata in file names

Commonly time.

#### 3. Metadata as attributes

The choice to distribute certain metadata fields as "attributes" can make sense when distributing file but makes little sense when considering the dataset as a whole, or as a cube. Example "ascending" and "descending" passes...

**insert more?**
> [name=Deepak Cherian] Yes! IIRC we had a nice list going somewhere


### Repository of untidy/ tidying examples

Link to github [repo](https://github.com/dcherian/tidy-xarray) with issue template (*in progress*)

> [name=Scott Henderson] Linking to short examples / visuals, especially of the common scenarios identified above would be great. Added an example here / ideas on what to put in a template https://github.com/dcherian/tidy-xarray/issues/7

[^1]: (heavily inspired from H. Wickham)