# RDF Data Cube Library

## Getting all dimensions, measures & attributes

To properly query the data we first need some basic metadata, to be able to generate a query for the observation the easiest way is to fetch all dimensions, attributes and measures for the datacube.

* dimensions are mandatory, in terms of every observation needs to have *all* dimensions
* measure(s) is the measure itself. It can be one or more. I think in our cubes it's always only one
* attributes: this is additional stuff that is related to the observation but can be optional. So it has to be OPTIONAL in SPARQL

This query works on https://lindas-data.ch/sparql-ui/ with all the `FROM` statements, use them to switch between different cubes

PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX qb: <http://purl.org/linked-data/cube#>
PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
SELECT *
# comment one of these FROMs to see what happens
FROM <https://linked.opendata.swiss/graph/FOEN/UBD28>
#FROM <https://linked.opendata.swiss/graph/FOEN/UBD66>
# in the two cases above you only get one dataset back.
# This one here has a whole bunch of them:
#FROM <https://linked.opendata.swiss/graph/zh/statistics> 
WHERE {
  ?dataset a qb:DataSet ;
           qb:structure/qb:component ?componentSpec .
  
  # kind: is it a dimension/attribute or measure
  # Note that you can basically filter on this URI, look up
  # dimension/attribute/measure on our service:
  # https://prefix.zazuko.com/qb:dimension
  ?componentSpec ?kind ?componentUri .
  # obviously the label could be multi-language, so we need to be
  # able to filter for that for the user
  ?componentUri  rdfs:label|skos:prefLabel ?label .
}

For Zurich we get a whole bunch of datasets back, in their documentation they show how to get a list of all datasets, plus labels, plus the amount of observations per dataset: https://github.com/StatistikStadtZuerich/documentation#list-of-available-datasets

If you would like to get the metadata for one particular dataset, just replace `?dataset` with the URI of the particular dataset. For `FOEN` graphs this won't make a difference as there is only one dataset per graph. In Zurich you will get far less dimensions back that way.

Example:

`<https://ld.stadt-zuerich.ch/statistics/dataset/AST-RAUM-ZEIT-BTA> a qb:DataSet ;`

## Fetching data

Each observation is stored as a `qb:Observation`, the most minimal query would simply fetch all observations in a store and we could start working with it. Obviously that's exactly what we do not want to do, as this might give back way too many data that way.

So as a next step we need to figure out the proper minimal query for the dataset we want to query. The minimal query would be:

?observation a qb:Observation ;
    <dimension1> ?uriOfDimension1 ;
    <dimension2> ?uriOfDimension2 ;
    <dimensionN> ?uriOfDimensionN ;
    

As mentioned before, for a specific DataCube, *all* dimensions need to be specified by an Observation.

In the query above we had a list of dimensions in the combination of `?kind` and `?componentUri`. If we would filter (or replace) `?kind` with `qb:dimension`, we would get all necessary `?componentUri`s back which we need to create a minimal query for observations. Example:

PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX qb: <http://purl.org/linked-data/cube#>
PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
SELECT *
# comment one of these FROMs to see what happens
#FROM <https://linked.opendata.swiss/graph/FOEN/UBD28>
#FROM <https://linked.opendata.swiss/graph/FOEN/UBD66>
# in the two cases above you only get one dataset back.
# This one here has a whole bunch of them:
FROM <https://linked.opendata.swiss/graph/zh/statistics> 
WHERE {
  <https://ld.stadt-zuerich.ch/statistics/dataset/BEW-RAUM-ZEIT> a qb:DataSet ;
           qb:structure/qb:component ?componentSpec .
  
  # kind: is it a dimension/attribute or measure
  # Note that you can basically filter on this URI, look up dimension/attribute/measure on our service https://prefix.zazuko.com/qb:dimension
  ?componentSpec qb:dimension ?componentUri .
  ?componentUri  rdfs:label|skos:prefLabel ?label . # obviously the label could be multi-language, so we need to be able to filter for that for the user
} LIMIT 10

So we filter on `qb:dimension` and hardcode the dataset to `<https://ld.stadt-zuerich.ch/statistics/dataset/BEW-RAUM-ZEIT>`.

We only get two dimensions back here: RAUM & ZEIT.

So a minimal query to get Observations of that DataSet back would be:

PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX qb: <http://purl.org/linked-data/cube#>
PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
SELECT *
FROM <https://linked.opendata.swiss/graph/zh/statistics> 
WHERE {
  ?observation a qb:Observation ;
    <https://ld.stadt-zuerich.ch/statistics/property/ZEIT> ?zeit ;
    <https://ld.stadt-zuerich.ch/statistics/property/RAUM> ?raum ;
    qb:dataSet <https://ld.stadt-zuerich.ch/statistics/dataset/BEW-RAUM-ZEIT> .
} LIMIT 10

Note that I added another restriction:

?observation qb:dataSet <https://ld.stadt-zuerich.ch/statistics/dataset/BEW-RAUM-ZEIT> ;

This is also mandatory, otherwise we will get any other Observation back from other DataCubes that also use RAUM & ZEIT as dimensions.

What we do not get back is the measure itself, so we have no idea what exactly was measured in this RAUM & ZEIT. If you replace `qb:dimension` in the query before with `qb:measure` you will get the URI of the measure for this dataset: `<https://ld.stadt-zuerich.ch/statistics/measure/BEW>`

So simply add this to the query for `qb:Observation`s.

Last but not least we might want to add all attributes. To get them properly we need to add each one of them as OPTIONAL. First we replace `qb:dimension` with `qb:attribute` to filter them, then we can generate the following, final query for this dataset:

PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX qb: <http://purl.org/linked-data/cube#>
PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
SELECT *
FROM <https://linked.opendata.swiss/graph/zh/statistics> 
WHERE {
  
  	?observation a qb:Observation ;
        <https://ld.stadt-zuerich.ch/statistics/property/ZEIT> ?zeit ;
        <https://ld.stadt-zuerich.ch/statistics/property/RAUM> ?raum ;
        <https://ld.stadt-zuerich.ch/statistics/measure/BEW> ?measure ;
        qb:dataSet <https://ld.stadt-zuerich.ch/statistics/dataset/BEW-RAUM-ZEIT> .
        
        OPTIONAL { ?observation <https://ld.stadt-zuerich.ch/statistics/attribute/QUELLE> ?attQuelle . }
        OPTIONAL { ?observation <https://ld.stadt-zuerich.ch/statistics/attribute/GLOSSAR> ?attGlossar . }
        OPTIONAL { ?observation <https://ld.stadt-zuerich.ch/statistics/attribute/FUSSNOTE> ?attFussnote . }
        OPTIONAL { ?observation <https://ld.stadt-zuerich.ch/statistics/attribute/DATENSTAND> ?attDatenstand . }
        OPTIONAL { ?observation <https://ld.stadt-zuerich.ch/statistics/attribute/ERWARTETE_AKTUALISIERUNG> ?attErwarteteAktualisierung . }
        OPTIONAL { ?observation <https://ld.stadt-zuerich.ch/statistics/attribute/KORREKTUR> ?attKorrektur . }

} LIMIT 10

Implementation note: `OPTIONAL` blocks in SPARQL can really kill performance so it might be a smart thing to have them, pun intended, optional in the library. Often they are not of interest for the users and only those providing the data really care about what is written in there.

## Fetching labels for dimensions

There are basically two types of dimensions (and to some extent attributes): Those that have a URI as "instance" of that particular dimension and those who have a literal (most probably a typed literal like `xsd:date`) as instance of the dimension. We can filter both in SPARQL so we can figure out if it's a literal or if it's a URI.

For literals we most probably will have to be able to filter for it. The most common literal is obviously dates so we need to be able to create this kind of `FILTER`s:

FILTER(?time = "2017-12-31"^^xsd:date)

This could also be any time range with `<` and `>`.

For the start I propose to simply add time based filtering and see where we go from there.

Spatial domains in the Zurich dataset are assigned as URI. For example we have the following URI used: `<https://ld.stadt-zuerich.ch/statistics/code/R30000>`. If we open this URI we see that this is `Stadt Zürich (ab 1934)`. In other words if we want to allow the user to filter for URI based dimensions, we need to fetch the labels from them and once the specific one(s) is selected, filter for it based on its URI. A simple filter would look like:

PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX qb: <http://purl.org/linked-data/cube#>
PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
SELECT *
FROM <https://linked.opendata.swiss/graph/zh/statistics> 
WHERE {
  
  	?observation a qb:Observation ;
        <https://ld.stadt-zuerich.ch/statistics/property/ZEIT> ?zeit ;
        <https://ld.stadt-zuerich.ch/statistics/property/RAUM> <https://ld.stadt-zuerich.ch/statistics/code/R30000> ;
        <https://ld.stadt-zuerich.ch/statistics/measure/BEW> ?measure ;
        qb:dataSet <https://ld.stadt-zuerich.ch/statistics/dataset/BEW-RAUM-ZEIT> .

} LIMIT 10

So basically we hard-code the URI of the object with the one for Zurichs city center itself.

This could also be done with a `FILTER`, even though this might be a bit less performant (depending on the query optimizer obviously):

FILTER( ?raum IN(<https://ld.stadt-zuerich.ch/statistics/code/R30000>))

Good thing of that notation is that we can specify multiple URIs, separated by `,`.

## Getting instances of a particular dimension

In the wild a bunch of predicates are common for labels, as you have seen in ontologies. In our cubes we use either `rdfs:label` or `skos:prefLabel`. These two should be supported for the moment and it should be easy to add new ones when necessary.

This query would fetch all referenced instances of dimension RAUM in a particular dataset:

PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX qb: <http://purl.org/linked-data/cube#>
PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
SELECT DISTINCT ?raum ?raumLabel
FROM <https://linked.opendata.swiss/graph/zh/statistics> 
WHERE {
  ?observation a qb:Observation ;
    <https://ld.stadt-zuerich.ch/statistics/property/RAUM> ?raum ;
    qb:dataSet <https://ld.stadt-zuerich.ch/statistics/dataset/BEW-RAUM-ZEIT> .
    
    ?raum rdfs:label|skos:prefLabel ?raumLabel .
}

So that could be used to offer filter lists to users.

## Getting min/max values for instances that are a literal

To be able to do proper rendering it's nice to know when a time range starts & ends without querying the full dataset. This query would provide this information for all dimension/measures/attributes:

PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX qb: <http://purl.org/linked-data/cube#>
PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
SELECT ?property (MIN(?propertyValue) AS ?min) (MAX(?propertyValue) AS ?max)
FROM <https://linked.opendata.swiss/graph/zh/statistics> 
WHERE {
        ?obs a qb:Observation ;
            qb:dataSet <https://ld.stadt-zuerich.ch/statistics/dataset/BEW-RAUM-ZEIT> ;
            ?property ?propertyValue .

        FILTER(isLiteral(?propertyValue))
} GROUP BY ?property

## Mapping to JS objects

Michael did quite some work regarding mapping SPARQL result sets to JS, please have a look at [d3-sparql](https://github.com/zazuko/d3-sparql) and either re-use or abstract this into a separate library.

There is one special case we did probably not take care of yet: There is no `null` value in RDF, expressing that something was measured but for whatever reason the value is not a number, we use `NaN`, originated in XML. In Zurich dataset this might be the case and in RDF it looks like this: https://discuss.rdf.community/t/discovering-rdf-data-cubes/75/3?u=ktk

# Next

If we can create this kind of queries, we in my opinion have a basic library available on which we can gather more ideas.

## Ideas

### Michael

- dimension continuous / not continuous (nominal dimension)
- ~~max / min of continuous (e.g. -10 to +200) or all values of not continuous (Altdorf, St. Gallen ...)~~