# EXAMS

Exams ([Eχαμs](https://github.com/mhardalov/exams-qa)) dataset is presenting a new benchmark for multilingual and cross-lingual question answering.
## Dataset Details

This dataset contains 24,143 questions with their choices in total from 16 languages. The questions are high-quality high school exam questions from subjects such as Natural Sciences, Social Sciences, Arts, etc.

Cross-lingual samples are obtained by gathering parallel examinations. Some exams in the dataset were offered in several languages in some countries. There are 9,857 parallel question pairs across seven languages.

The structure of the questions is as given in the samples. Stem of the question is given as a string and choices are a list with their labels such as "A", "B". Correct answers are given in 'answerKey', and 'info' contains the grade, subject, and language information.

You can reach the [ACL Paper](https://aclanthology.org/2020.emnlp-main.438.pdf) and [GitHub repositories](https://github.com/mhardalov/exams-qa) of this dataset via given links.

**Version**: This version of the dataset is retrieved from the original github repository at 2022.4.21 from commit id: ```f859e66```

### Samples

{
"id": "1f556343-8571-11ea-a5d8-54bef70b159e",
"question": 
    {"stem": "Aşağıdakilerden hangisi, Birleşmiş Milletler Genel Kurulunda, 9 Aralık 1948 tarihinde “Soykırım Suçunun Önlenmesi ve Cezalandırılmasına İlişkin Sözleşme”nin kabul edilmesinin nedenlerinden biridir?",
    "choices": [
        {"text": "ABD’nin Vietnam Savaşı’nda izlediği politikaya Batılı müttefiklerin destek vermemesi", 
        "label": "A"},
        {"text": "Filistinlilerin ülkelerinden çıkarılmalarının da etkisiyle Mısır-İsrail Savaşı’nın yaşanması", 
        "label": "B"},
        {"text": "II. Dünya Savaşı sırasında hukuki olmayan ve insanlık dışı uygulamaların yaşanması",
        "label": "C"}, 
        {"text": "Afrika’da şiddetli etnik ayrımcılığın ve düşmanlığınhüküm sürmesi",
        "label": "D"}, 
        {"text": "Yugoslavya’nın dağılma sürecinde Sırp milliyetçilerinin Boşnaklara soykırım uygulaması", 
        "label": "E"}]}, 
"answerKey": "C", 
"info": {
    "grade": 12, 
    "subject": "History",
    "language": "Turkish"}
}

### Fields

Explain the fields of the instances.

| field | dtype |
|----------|------------|
| id | string |
| question | dictionary|
| answerKey | string|
| info | dictionary|

### Splits

Indicated multilingual train/validation/test split sizes for each language.

|Language | Training| Validation | Test |Cross-lingual Training| Cross-lingual Dev|
|---------|---------|------------|-------|--|--|
|Albanian |565 | 185 | 755 |1194|311|
|Arabic | - |- |562|-|-|
|Bulgarian| 1100| 365 | 1472|2344|593|
|Croatian |1003| 335| 1541|2341|538|
|French |-|-|318|-|-
|German |- | - | 577|-|-|
|Hungarian| 707|263|1297|1731|536|
|Italian |464|156|636|1010|246|
|Lithuanian| -|-|593|-|-|
|Macedonian| 778|265|1032|1665|410
|Polish | 739|246|986|1577|394|
|Portuguese|346|115|463|740|184|
|Serbian|596|197|844|1323|314
|Spanish|-|-|235|-|-|
|Turkish|747|240|977|1571|393|
|Vietnamese|916|305|1222|1955|488|
|Combined | 7961|2672|13510|-|-|

### Subject Analysis

There are 3 general subjects in the dataset given as Natural Science, Social Science, and others. The sub topics are listed below.

**Natural Science**: Biology, Chemistry, Geology, Physics, Science.

**Social Science:** Business and Economy, Citizenship, Ethics, Geography, History, Philosophy, Politics, Psychology, Social, Sociolohy.

**Others:** Agriculture, Fine Arts, Forestry, Informatics, Islamic Studies, Landscaping, Professional, Religion, Tourism.

## Dataset Creation

### Curation Rationale

The dataset is created to present a benchmark for multilingual and cross-lingual question answering studies.

### Data Source

Exams dataset is collected from official state exams prepared by the ministries of education of several countries.

### Annotations

The questions are written and answered by educational experts. In addition, the questions have been cleaned by manually reviewing each question to verify symbols and text were displayed correctly.

### Quality

The questions and answers are from verified sources such as ministries of education of several countries.

## Considerations

### Social Impact of Dataset

This dataset is part of an effort to encourage question answering studies in several languages and cross-lingual knowledge transfering.

## Additional Information

### Dataset Curators

Published by Momchil Hardalov, Todor Mihaylov, Dimitrina Zlatkova, Yoan Dinkov, Ivan Koychev, Preslav Nakov.

### Citation Information

Please cite the following paper if you found this dataset useful.

[[1]](https://aclanthology.org/2020.emnlp-main.438/) M. Hardalov, T. Mihaylov, D. Zlatkova, Y. Dinkov, I. Koychev, P. Nakov "EXAMS: A Multi-subject High School Examinations Dataset for Cross-lingual and Multilingual Question Answering", Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020.

@inproceedings{hardalov-etal-2020-exams,
    title = "{EXAMS}: A Multi-subject High School Examinations Dataset for Cross-lingual and Multilingual Question Answering",
    author = "Hardalov, Momchil  and
      Mihaylov, Todor  and
      Zlatkova, Dimitrina  and
      Dinkov, Yoan  and
      Koychev, Ivan  and
      Nakov, Preslav",
    booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing",
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.emnlp-main.438",
    pages = "5427--5444",
    series = "EMNLP~'20"
}