---
title: Wikidata as authority linking hub
lang: en
slideOptions:
README: https://github.com/hakimel/reveal.js/#configuration
slideNumber: false
showNotes: false
theme: white
---

<!-- Adjust the default reveal.js style -->
<style type="text/css">
@import "https://hackmd.io/build/reveal.js/css/theme/white.css";
.reveal h1, .reveal h2, .reveal h3 { text-align: left; text-transform: none }
.reveal p, .reveal ul, .reveal ol { text-align: left; display: block; }
.reveal section img { border: none; box-shadow: none; }
</style>

<!--
Unter https://hackmd.io/p/S1YmXWC0e werden die Slides im Präsentationsmodus mit reveal.js angezeigt. Mit "S" werden die Speaker-
s angezeigt.
-->

# Wikidata as authority linking hub

*Joachim Neubert (ZBW - Leibniz Information Centre for Economics, Kiel/Hamburg)*
*Jakob Voß (Verbundzentrale des GBV, Göttingen)*

ELAG
Athens, June 8, 2017

`Online version (with links): https://hackmd.io/p/S1YmXWC0e`

---

# Introduction

---

## Authority files

Consistently refer to entities

* Via identifier ("things, not strings")
* GND, MeSH, STW, ISIL, RePEc Author ID...

---

## Linking hubs

Connect identifiers among authority files

* `owl:sameAs`, `skos:exactMatch`, `skos:closeMatch`...
* [VIAF](http://viaf.org), sameAs.org, [Wikidata](https://wikidata.org)...


Note: Two general models: 1:1 or hub. Differences of hubs
* VIAF - Libraries / algorithmically generated clusters
* samAs - Semantic Web / harvested coreference links from the web
* wikidata - Community / curated content

---

## Wikidata

* Knowledge base of Wikimedia projects
* All kinds of entities
* concepts, places, people, works...

---

## Wikidata usage

* Editable by anyone
* via Website and API
* via apps that use the API
* Data available
* http://query.wikidata.org/ (SPARQL)
* JSON API & database dumps

---

## Wikidata statements


Note: Wikidata consists of
* Items
* uniquely identified by an item-id (Q...)
* having a label in each language
* Statements
* each with a property
* uniquely identified with a property-id (P...)
* and a value (string, date, item, external identifier...)
* and optional qualifiers and references to sources

---

## Wikidata item example


Note: on the example
* This is a more condense display of a Wikidata item, based on the "SQID" Wikidata browser.
* Note the list of identifiers for this item/person, linking to other authority files.
* Property URL pattern + inserted ID value = link

---

## Authority file identifiers in Wikidata

More than half of all Wikidata properties

* Datatype external identifier (~1,750)
* [Properties for authority control](https://www.wikidata.org/wiki/Q18614948) (~1,500)
* Properties with corresponding KOS (~220)

Note: The total number is hard to measure because of the "organic" organization of Wikidata

---

# Wikidata---ISIL (organizations)

Example:

*Neuschwanstein Castle* ([Q4152](https://www.wikidata.org/wiki/Q4152))
ISIL ([P791](https://www.wikidata.org/wiki/Property:P791)): *DE-MUS-051612*

Current state:

* lobid.org: ~15,000 ISIL (DACH only)
* Wikidata: ~6,500 ISIL

---

## Tool: *Mix'n'match*

* Web application mapping tool
* Helps to add 1-to-1-mappings

https://tools.wmflabs.org/mix-n-match/

---

## Step 1: Upload ISIL list with names


---

## Step 2: Confirm match candidates


---


Note: The visual mode is even more handy to confirm mappings.
The ISIL-Mapping is finished for Geman Museums (lobid-museums).

---

# GND---RePEc Author ID

* In the [EconBiz](http://www.econbiz.de) economics search portal authors are identified differently:
* by **GND ID** in data from ZBW's *Econis* catalog
* by **RePEc Author ID** in data from *Research Papers for Economics*
* Large volumes: 450,000 vs. 50,000 distinct persons
* ~3,000 pairs of IDs discovered in a previous project

Note: Characteristics
* GND = well known and interlinked
* RAS = high qualitiy data, curated by the authors themselves, incented by rankings - but: linked to nothing else in the world

---

### Person identifiers in Wikidata and EconBiz


Note: Maintaining a custom mapping environment would require
* custom software, database, operations
* access limited to ZBW staff

---

## Utilizing Wikidata as linking hub

* Wikidata-Properties for both identifier systems
* GND ID ([P227](https://www.wikidata.org/wiki/Property:P227)): ~375,000 items which are humans
* RePEc Short-ID ([P2428](https://www.wikidata.org/wiki/Property:P2428)): ~2,200 items
* Since every identifier should identify exactly one person, we can derive
* GND ID ⟶ Wikidata ID ⟶ RePEc ID
* RePEc ID ⟶ Wikidata ID ⟶ GND ID

where both properties have values (~760 items, as of 2017-04-25)

Note: I describe the process in 6 steps, which should be portable to other similar mappings. All numbers above as of April 25th 2017

---

## Step 1: Supplement WD items with RePEc Author IDs

* 77 WD items with GND ID without RePEc Short-ID

* Transform to *[QuickStatements2](https://tools.wmflabs.org/quickstatements/)* input file ([SPARQL query](https://github.com/zbw/repec-ras/blob/ELAG2017/sparql/missing_ids_in_wikidata_from_mapping.rq), [script](https://github.com/zbw/repec-ras/blob/ELAG2017/bin/create_missing_ids_in_wikidata_from_mapping.pl))

* Copy & paste to *QuickStatements2*

---

### Bulk editing with *QuickStatements2*


Further simplification with upcoming release of *[wdmapper](https://wdmapper.readthedocs.io/)* command line tool

---

## Step 2: Supplement WD items with GND IDs

* 384 WD items with RePEc Short-ID without GND ID
* *same process as other direction*

---

## Step 3: Add "most important" authors with RePEc identifiers

* Scraped from ranking pages ([Top 10% economists](https://ideas.repec.org/top/top.person.all.html), [Top 10% female economists](https://ideas.repec.org/top/top.women.html))
* Transform and load into *Mix'n'match* ([RePEc Top](https://tools.wmflabs.org/mix-n-match/#/catalog/406), [RePEc Top Female](https://tools.wmflabs.org/mix-n-match/#/catalog/404))
* *Same process as ISIL use case*
* Confirm match candidates (~ 1,600 of 4,600)

---

## Step 4: Add "most important" authors with GND identifiers

* 18,000 GND authors with >30 publications in EconBiz
* Transform and load into *Mix'n'match* ([GND economists (de)](https://tools.wmflabs.org/mix-n-match/#/catalog/431))
* 25% matched automatically with Wikidata items
* Confirm match candidates (~ 1500)

---

## Step 5: Rinse and repeat

* Repeat *Mix'n'match* "sync" operation before starting to work manually
* often, people are adding data at fast rate!
* Repeat bulk adding of missing identifiers to make use of complementing identifiers added meanwhile

---

## Step 6: Add missing Wikidata items

* Verify missing authors indeed are not in Wikidata (step 3 and 4)
* Generate Wikidata item data from existing mappings (or lists, e.g., top female economists) in *QuickStatements2* input format ([SPARQL query](https://github.com/zbw/repec-ras/blob/ELAG2017/sparql/ras_missing_in_wikidata.rq), [script](https://github.com/zbw/repec-ras/blob/ELAG2017/bin/create_missing_wikidata.pl))

<!--  -->

---

### Synthezised Wikidata items from ZBW's GND--RePEc mapping

2179 new Wikidata items created
* occupation "economist" from RePEc Author Service
* gender and date of birth/death (if available) from GND
* description from GND's info field and RePEc's "works for"

---

### Recommendations for item creation

* Explain your plan and ask for feedback in the [Wikidata project chat](https://www.wikidata.org/wiki/Wikidata:Project_chat)
* Pay attention to [Wikidata's notability criteria](https://www.wikidata.org/wiki/Wikidata:Notability)
* [Apply for a bot account](https://www.wikidata.org/wiki/Wikidata:Requests_for_permissions/Bot) to make mass edits ([example](https://www.wikidata.org/wiki/Wikidata:Requests_for_permissions/Bot/JneubertAutomated_2))
* Use *QuickStatements2* (more convenient input format, batch mode, sources)
* Source every statement ([hints](https://www.wikidata.org/w/index.php?title=Wikidata:Project_chat/Archive/2017/05#Source_statements_for_items_synte

d_from_authorities_-_recommendations.3F))

---

## Result: the mapping in Wikidata

* 4087 matching GND - RePEc author IDs
* 3081 matches from ZBW's mapping
* 1006 matches contributed by Wikidata users
* 468 additional RePEc author IDs
* 52,444 additional GND IDs (present in EconBiz)

(as of 2017-06-04)

Note: This looks even better if we focus on the "most important" persons.

---

### Top 10% RAS and frequent (>30) GND in EconBiz


Note: The mapping currently covers more than 60% of RePEc's Top 10% Economists.

---

### Further results

* Mappings to other authorities *for free* (e.g., currently ~1550 RePEc ⟷ VIAF)
* Identifiers and items inserted by individual Wikidata contributors add up continuously
* Mapping steps can be repeated with additional data (e.g., [top economists from Latin America](https://ideas.repec.org/top/top.lamerica.html#authors))
* Further identifiers (VIAF, ORCID...) provide more opportunities for indirect matching

Results from _every step in the mapping process_ and all indiviual efforts are immediately available

---

# Tools

* *Mix'n'match* (system-supported intellectual matching)
* *QuickStatements2* (addition of generated properties and items)
* [*wdmapper*](https://github.com/gbv/wdmapper) (harvest, diff & add mappings)
* Support of indirect mappings (e.g., GND-WD-RePEc) in one step
* Work in progress (no adding by now)
* Daily harvested mappings in multiple formats:
http://coli-conc.gbv.de/concordances/wikidata/

Tools for mass editing require approved bot account.

---

## Limitations

* Mapping algorithms to find mapping candidates
* Limitation to easy-1-1-relationships
* part-whole
* often new Wikidata items required
* depends on the use case
* Large sets of mappings and results
* Regular review required for maintainance

Note:
* We just relied on the basic mapping algorithm build into Mix'n'match and added already existing mappings to Wikidata
* Each tool has its limitation on large result sets

---

# Benefits

* Outsourced interface, storage, and operation
* Crowdsourced mapping maintenance
* Wikidata has policies and tools for data quality
* Open Data for multiple and unknown uses
* Additional benefits:
* multilingual Wikipedia links
* lots of (formatted) data, nice pictures, ...
* links to multiple other vocabularies

---

# Contact

Joachim Neubert
[e-mail]
http://zbw.eu/labs


Jakob Voss
[e-mail]
http://jakoblog.de/