# SuperGLUE
[A Stickier Benchmark for General-Purpose Language Understanding Systems](https://w4ngatang.github.io/static/papers/superglue.pdf)

GLUE原先的任務只包含sentence pair的分類，SuperGLUE增加了指代消集與QA

# SuperGLUE任務:

| 語料 | Train | Dev | Test | 任務 | 資料來源 |
| ------- | ----- | ---- | ---- | ----------- | --- |
| BoolQ | 9427 | 3270 | 3245 | QA |google queries, Wikipedia |
| CB | 250 | 57 | 250 | NLI |various |
| COPA | 400 | 100 | 500 | QA |blogs, photographt encyclopedia |
| MultiRC | 5100 | 953 | 1800 | QA |various |
| ReCoRD | 101k | 10k | 10k | QA |news(CNN, DailyMail) |
| RTE | 2500 | 278 | 300 | NLI |news, Wikipedia |
| WiC | 6000 | 638 | 1400 | WSD |WordNet, VerbNet, Wiktionary |
| WSC | 554 | 104 | 146 | coreference |fiction books |

*WSD(word sense disambiguation 單詞歧義消除)

## [BoolQ(Boolean Question)](https://arxiv.org/abs/1905.10044)
input : Context + Question
output : Answer(Yes/No)

## [CB(CommitmentBank)](https://semanticsarchive.net/Archive/Tg3ZGI2M/Marneffe.pdf)
判斷說話的人對某一事件的承認程度，"真實", "不真實", "不確定"
input : 兩句的context+一句target組成
output : -3~3的分數

## [COPA(Choice of Plausible Alternatives)](https://ict.usc.edu/pubs/Choice%20of%20Plausible%20Alternatives-%20An%20Evaluation%20of%20Commonsense%20Causal%20Reasoning.pdf)
input : Premise, Question, Alternative 1, Alternative 2
output : Correct Alternative

## [MultiRC(Multi-Sentence Reading Comprehension)](https://cogcomp.seas.upenn.edu/page/publication_view/833)
多重選擇題
input : context, question, alternative
output : answer

## [ReCoRD(Reading Comprehension with Commonsense Reasoning Dataset)](https://arxiv.org/abs/1810.12885)
input : context, query(query會挖空格)
output : 預測空格要填入什麼

## [RTE(Recognizing Textual Entailment)](https://openreview.net/pdf?id=rJ4km2R5t7)
GLUE的其中一項困難任務，所以被保留
RTE1 + RTE2 + RTE3 + RTE5
二分類任務 entailment/not_entailment
受益於transfer learning

## [WiC(Word-in-Context)](https://arxiv.org/abs/1808.09121)
input : 兩段context + 兩句中重複出現的詞
output : 是否意義相同

## [WSC(Winograd Schema Challenge)](https://cs.nyu.edu/faculty/davise/papers/WinogradSchemas/WS.html)
指代消集問題