# OpenRefine-SDC integration: design questions for the SDC team

## Background

[OpenRefine](https://openrefine.org/) is a data wrangling tool used by various communities to transform data into structured formats, using the Extract-Transform-Load (ETL) paradigm.

It is used by the Wikidata community for data imports thanks to the [dedicated integration](https://www.wikidata.org/wiki/Wikidata:Tools/OpenRefine) for this knowledge graph in the tool as an extension that is shipped in the official bundle. ([Short video tutorial for the integration](https://www.youtube.com/watch?v=wfS1qTKFQoI).)

In the summer 2020, Lu Liu (GSoC student working on OpenRefine) is working on generalizing the extension to make it compatible with any Wikibase instance ([OR issue #1640](https://github.com/OpenRefine/OpenRefine/issues/1640)). As part of this work, we need to come up with a way to describe a Wikibase instance and all the surrounding services that OpenRefine relies on for its integration (SPARQL query service, reconciliation service and a large number of configuration parameters).

This document presents some questions from the OpenRefine team about Structured Data on Commons, which come up as we draft plans to add support for it in OpenRefine ([OR issue #2144](https://github.com/OpenRefine/OpenRefine/issues/2144)).

Wikidata integration in OpenRefine is based on the [Wikidata-Toolkit Java library](https://github.com/Wikidata/Wikidata-Toolkit), which we actively contribute to, and some of the generalizations we need to do will happen in Wikidata-Toolkit itself (such as [WDTK issue #474](https://github.com/Wikidata/Wikidata-Toolkit/issues/474)), therefore benefiting a range of other tools.

## Our goal with SDC integration

We want to make this integration **as generic as possible**: just like our Wikidata support should have been generic enough to support any Wikibase instance from the outset, we want to make sure we anticipate any generalization of SDC's set up as early as we can.

## Main issues

### Federation

Wikimedia Commons uses Wikidata entities in its statements. Our understanding is that although this mechanism is readily available to other Wikibase instances, it is currently restricted to MediaWiki instances sharing the same database cluster (so federation between a WMF Wikibase and a third-party Wikibase is not possible yet).

The way we represent Commons' federation scenario with Wikidata in OpenRefine should ideally match any plans you have on your side to make this mechanism more generic and reusable. If other third party Wikibase instances get federated, it should only be a matter of changing OpenRefine's configuration to upload structured data to other instances which use federation.

WMDE's [federated properties project](https://phabricator.wikimedia.org/tag/wikibase_-_federated_properties/) is interesting for us in this regard. To configure a Wikibase instance in OpenRefine, we could of course replicate the configuration options available to Wikibase instance administrators introduced in this project. But as we anticipate those configuration options might become more flexible in the future, it could be useful to have a broad idea of the range of likely generalizations and come up with an integration that would easily adapt to these.

For instance, does it make sense for us to build our integration with the intention to support:
* mixes of local and federated properties in the same entity?
* snaks with local properties and federated values?
* snaks with federated properties and local values?
* … a lot more scenarios could be considered!

With these sort of questions in mind, it should be possible to come up with a configuration format which is flexible enough for a reasonable range of federation scenarios. The technical hurdles to federation are much lower on OpenRefine's side, since we are already used to representing Wikibase entities through the view of the MediaWiki API, so it is mostly a question of getting the expressivity of the configuration format right. So even if some scenarios currently seem hard to add support for from Wikibase's technical perspective, it might be worth keeping them in scope for us if they make sense as a legitimate user story for Wikibase.

What we need to understand on our side is what to expect as a MediaWiki API client: for instance, being able to determine reliably which Wikibase instance an entity id is coming from, when consuming statements in JSON, or knowing which Wikibase instance holds the domain of snak values for a given property, and so on.

### Entity id generation

Commons uses the page ids of the media files to associate MediaInfo entity ids to them. If another MediaWiki instance wanted to add structured data to the files it holds, would it necessarily use the same mechanism? If not, what would be the appropriate MediaWiki API action to convert a file name to an entity id or conversely?

### Discoverability of Wikibase's configuration

OpenRefine needs to know of quite a few of Wikibase's configuration parameters to integrate with it:

* MediaWiki namespaces and their associated page types (wikitext, file, item, property, lexeme, mediainfo…);
* RDF namespaces;
* Special entities which have a custom role (such as [formatter URL (P1630)](https://www.wikidata.org/wiki/Property:P1630));
* Entity ids used by Wikibase Quality Constraints, if installed (such as [property constraint (P2302)](https://www.wikidata.org/wiki/Property:P2302));
* Query service URL;
* and so on.

All this can be discovered by humans who can cook up a configuration file for OpenRefine with the appropriate parameter values.

However, as a Wikibase admin, I would ideally like to configure these parameters in a single location (Wikibase's own config file) and tools such as OpenRefine should be able to retrieve them from Wikibase directly (via some action in the MediaWiki API).

This is not a strong requirement for the integration: just something worth keeping in mind when adding new configuration parameters that could be safely exposed.

This problem has been tackled by WMDE in their Manifest Extension project:
https://phabricator.wikimedia.org/tag/wikibase_-_automated_configuration_detection_wikibasemanifest/