---
# System prepended metadata

title: 'Notes pour le cours de Biais dans les collections numériques : données et représentation(s)'

---


# Notes pour le cours de Biais dans les collections numériques : données et représentation(s)

[toc]

Link Zoom: [https://unige.zoom.us/j/67032113169?pwd=OU9GZGdqeWp2RDdVRE9ZUzdIWDJaQT09](https://unige.zoom.us/j/67032113169?pwd=OU9GZGdqeWp2RDdVRE9ZUzdIWDJaQT09)



## Dataset

### Links

To download the full dataset use this link: [Download CSV ULAN](https://drive.switch.ch/index.php/s/RUDkAEPJtidc828)  

#### Splitted datasets:

- Pauline: [link](https://drive.switch.ch/index.php/s/RowqJsdcHlkdmIz)
- Frédérine: [link](https://drive.switch.ch/index.php/s/ZPpGMcsCW2BaqLg)
- Marie: [link](https://drive.switch.ch/index.php/s/Kek0paeNo0CS7SR)
- Rosie: [link](https://drive.switch.ch/index.php/s/2uYQkGHCkfPi9KB)

### Nationalities to Country
For reconciling nationalities to the corresponding country we used [this dataset](https://drive.switch.ch/index.php/s/9N1SZ2tUqEJOHey) derived from [wikipedia](https://en.wikipedia.org/wiki/List_of_adjectival_and_demonymic_forms_for_countries_and_nations) .

### Creation

The dataset has been produced as result of the following query

```sparql
PREFIX gvp: <http://vocab.getty.edu/ontology#>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX foaf: <http://xmlns.com/foaf/0.1/>
PREFIX schema: <http://schema.org/>

SELECT ?artist ?term (lang(?term) as ?lang) ?birth ?death ?deathPlace ?labelDeath ?birthPlace ?labelBirth ?description ?nationality ?termNationality ?gender ?genderLabel WHERE {
  
  
  ?artist rdf:type gvp:PersonConcept ;
           gvp:prefLabelGVP ?preflabel .
  ?preflabel gvp:term ?term .


  OPTIONAL {
  ?artist foaf:focus ?focus .
  ?focus  gvp:biographyPreferred ?bio ;
        }

  OPTIONAL {      
  ?bio gvp:estStart ?birth .
        }

  OPTIONAL {
  ?bio gvp:estEnd ?death .
        }

  OPTIONAL {
   ?bio schema:birthPlace ?birthPlace .

   ?BirthPlaceFull  foaf:focus ?birthPlace ;
                    gvp:prefLabelGVP ?prefLabelPlaceBirth .

  ?prefLabelPlaceBirth gvp:term ?labelBirth .
        }

  OPTIONAL{
    ?bio schema:deathPlace ?deathPlace .
    ?deathPlaceFull foaf:focus ?deathPlace ;
                  gvp:prefLabelGVP ?prefLabelPlaceDeath .
    ?prefLabelPlaceDeath gvp:term ?labelDeath .
        }

  OPTIONAL{
    ?bio schema:description ?description .
        }

  OPTIONAL{     
  ?focus gvp:nationalityPreferred ?nationality .
  ?nationality gvp:prefLabelGVP ?preflabelNationality .
  ?preflabelNationality gvp:term ?termNationality .
        }

  OPTIONAL{
  ?bio schema:gender ?gender .
  ?gender gvp:prefLabelGVP ?genderTerm .
  ?genderTerm gvp:term ?genderLabel .
        }

  FILTER (?artist != <http://vocab.getty.edu/ulan/500775627>)
  
  }
```

## Analysis

### Links

Link to download: [https://nlp.stanford.edu/software/stanford-corenlp-latest.zip](https://nlp.stanford.edu/software/stanford-corenlp-latest.zip)

### Operation

#### Steps

##### nationalities

For each nationalities indicated in the dataset add a corresponding modern country:


1. retrieve the corresponding country from each nationality and store them in a new column called "Country"
	- use cell.cross to retrieve corresponding countries: `cell.cross("nationalities", "adj")[0].cells["country"].value`
		- Open the ULAN dataset on openrefine and name it accordingly
		- Download and import in openrefine the already-made nationalities table available here: https://docs.google.com/spreadsheets/d/10rGf_D1DI0xc1Xa0txP48LSnZxFbNWgpTLloVoCu5g0/edit?usp=sharing
		- use the formula cell.cross (documented here: https://docs.openrefine.org/manual/grelfunctions) in the ULAN dataset (column "termNationality") to create a new column
			- The formula should match the value in the column "termNationality" with the values present in the column "adj" in the nationalities table. For each match, the formula shoud extract the corresponding value in the "country" column on the nationalities tables.
2. Not everything has a match. Analyse the records without a match and add the modern corresponding country in the column "Country". 
3. Not everything has a direct modern corresponding country. Assign one when possible (use also the birth place information when needed)
	- Create a new column and use it for tagging, whenever needed, the entries with the appropriate vocabulary terms available at https://hackmd.io/4w_av1SATzmyEtS4Ra_vlA#Vocabulary-uncertainty . The terms can be expanded if needed

##### professions

For each description extract the relative professions

1. Retrieve the profession using NLP from the column "description".
	 - Download and install Stanford Core NLP [here](https://hackmd.io/4w_av1SATzmyEtS4Ra_vlA#Operation)
	 - Run the application in the terminal using the command (from the directory where Core NLP is installed):  `java -mx4g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer -port 9000 -timeout `
	 - Start Openrefine and add a "new column based on this column" from the column "description"
	 - Select Python/JYthon as language on Openrefine and insert the following code to query the NLP endpoint:

```python=
import urllib2
url = 'http://localhost:9000/?properties=%7B%22annotators%22%3A%22tokenize%2Cner%22%2C%22outputFormat%22%3A%22xml%22%7D'
data = value
post = urllib2.urlopen(url, data)
return post.read()
```


- Let it work. In case of errors, repeat the operation selecting (facet) a smaller (between 5000 and 10000) number of records
	- The process will result in a new column containing XML elements describing the professions and other information retrieved from the value in the description column
	- Extract the profession using the following code in Openrefine (always with Python/Jython language selected):

```python=
from xml.etree import ElementTree as ET
element = ET.fromstring(value)

resultsList = element.findall(".//token")

for result in resultsList:
    typerecord = result.find("NER")
    if typerecord.text == "TITLE":
        return result.find("word").text		
```


#### NLP

Link to download: [https://nlp.stanford.edu/software/stanford-corenlp-latest.zip](https://nlp.stanford.edu/software/stanford-corenlp-latest.zip)

Run with 
```bash
java -mx4g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer -port 9000 -timeout 15000
```

Code python for OpenRefine 

```python
import urllib2
url = 'http://localhost:9000/?properties=%7B%22annotators%22%3A%22tokenize%2Cner%22%2C%22outputFormat%22%3A%22xml%22%7D'
data = value
post = urllib2.urlopen(url, data)
return post.read()
```

Code python for extracting values in OpenRefine

```python
from xml.etree import ElementTree as ET
element = ET.fromstring(value)

resultsList = element.findall(".//token")

for result in resultsList:
    typerecord = result.find("NER")
    if typerecord.text == "TITLE":
        return result.find("word").text
```

### Issues

Not everything is a nationality. Some are cultures and not nationalities

### Vocabulary uncertainty

To use these terms for indicating problematic nationalities

- historical (e.g. Byzantine)
- not a nationality (e.g. international)
- to verify
- ambiguous nationality
- disputed region

### Unusable subjects

List of unusable subjects for our analysis.


1. `http://vocab.getty.edu/ulan/50077562` = it is named “NEW SUBJECT” and does not contain relevant information
2. `http://vocab.getty.edu/ulan/50045517` = museum. It represent a **museum** and not a person  
3. `http://vocab.getty.edu/ulan/500353456` = horse. It represent a **horse** and not a person  

