---
# System prepended metadata

title: Silberzahn et al. (2018) 領讀摘要
tags: [TCPSR, Reproducibility]

---

---
title: Silberzahn et al. (2018) 領讀摘要
tags: TCPSR,Reproducibility
description: View the slide with "Slide Mode".
---

## Silberzahn et al. (2018) <br> 領讀摘要

---

## 這項研究的價值

<!-- Put the link to this slide here so people can follow -->
- 心理科學史上第一次群眾外包資料分析專案
- 近期關注度最高的案例：Neuroimage data analysis [(Botvinik-Nezer et al., 2020)](https://www.nature.com/articles/s41586-020-2314-9)
    - 參考[TCPSR 線上研讀會#6](https://youtu.be/UNcRkVXy9jE)

---

## 啟動這項研究的原因

<small>[總主持人公開召募信](https://docs.google.com/document/d/1uCF5wmbcL90qvrk_J27fWAvDcDNrO9o_APkicwRkOKc/edit)</small>

- 一篇論文呈現一種分析方法與結果，實際上只有研究者知道跑了多少種。
- 研究者跑了多種分析，再以主觀挑選符合預期的分析結果，並不符合嚴謹的科學方法論。
- 不同組研究者針對同一個假設，分析同一筆資料，會有多少種分析方法？結果差異有多大？

---

## 假設與資料來源

- 職業足球比賽皮膚黑的球員會比皮膚白的球員收到較多的紅牌嗎？
- [2012-2013歐洲四國職業足球比賽紀錄](https://osf.io/47tnc/)

---

### 工作階段

|<small>階段</small>|<small>工作目標</small>|<small>學習重點</small>|
|:---:|:---|:---|
|<small>1</small>|<small>建立資料集</small>|<small>編輯資料編碼簿</small>|
|<small>2</small>|<small>招募小組；工作前調查</small>|<small>調查項目相當於預先註冊</small>|
|<small>3</small>|<small>第一輪小組分析工作</small>||
|<small>4</small>|<small>小組成果輪流互評</small>|<small>相當於初次正式投稿</small>|
|<small>5</small>|<small>第二輪小組分析工作</small>||
|<small>6a</small>|<small>開放討論；總體分析</small>|<small>相當於修改後再提交</small>|
|<small>6b</small>|<small>撰寫總報告</small>||
|<small>7</small>|<small>正式投稿期刊；小組外專家評審</small>||


---

## (預先註冊)小組工作前調查

- 13題必答、可附上示範腳本。
- 瀏覽[問卷項目](https://hackmd.io/_6NPEJXfTZG4oWiYx2KwqQ?view#%E5%B7%A5%E4%BD%9C%E5%89%8D%E8%AA%BF%E6%9F%A5%E5%95%8F%E5%8D%B7)
- 33個小組回覆；29個小組完成最後報告。

---

## 全部分析結果

![](https://fivethirtyeight.com/wp-content/uploads/2016/03/truth-vigilantes-soccer-calls2.png?w=2048)
<small>source: [(Aschwanden, 2016)](https://fivethirtyeight.com/features/failure-is-moving-science-forward/)</small>

---

## 各小組認為資料支持假設的可能性評估

![](https://journals.sagepub.com/na101/home/literatum/publisher/sage/journals/content/ampa/2018/ampa_1_3/2515245917747646/20181024/images/large/10.1177_2515245917747646-fig4.jpeg)
<small>source: [Silberzahn et al. (2018) Figure 4](https://journals.sagepub.com/doi/figure/10.1177/2515245917747646?)</small>

---

## 對QRP的反思

- p-hacking: 看到正面證據的期待 -> 不斷嘗試逼出顯著結果
- 歧路花園: 儘可能做各種可行的分析 -> 執行與假設最一致的分析
- 評審偏好/偏誤: 評審與原作者意見不一致的創新解方？

---

## 延伸閱讀

> Bishop, D. V. (2020). The psychology of experimental psychologists: Overcoming cognitive constraints to improve research: The 47th Sir Frederic Bartlett Lecture. Quarterly Journal of Experimental Psychology, 73(1), 1–19. doi: 10.1177/1747021819886519

---

## 衍生問題

1. 今天準備要開始一項新研究，你如何決定分析方法？
2. 如果分析結果不如預期，你有怎麼樣的處理經驗？
3. 最後討論到的三個QRP議題，你曾有相關的經驗嗎？

<small>請在[紀錄共筆](https://hackmd.io/RPooc0rURJiUjWN33j6Qlg)摘要個人與彼此的意見</small>

---

##### 工作前調查問卷

What transformations (if any) were applied to the variables. Please / be specific. 	

Were any cases excluded, and why?


What is the name of the statistical technique that you employed?	

Please describe the statistical technique you chose in more detail. / Be specific, especially if your choice is not one you consider to / be well-known. 	

What are some references for the statistical technique that you / chose?	

Which software did you use? If you used multiple kinds, please / indicate what was accomplished with each piece of software (e.g., / Data cleaning - R; Model estimation - SAS)	What distribution did you specify for the outcome variable of red / cards?	

What variables were included as covariates (or control variables) / when testing research question 1: The relationship between player skin tone and red cards / received?	

What variables were included as covariates (or control variables) / when testing research question 2a: The relationship between referee country implicit skin-tone prejudice and red cards received by...	

What variables were included as covariates (or control variables) / when testing research question 2b: The relationship between referee country explicit skin-tone prejudice and red cards received by...	

What theoretical and/or statistical rationale was used for your /  choice of covariates included in the models?

What unit is your effect size in?	

What is the size of the effect for research question 1: The relationship between player skin tone and red cards / received?  /   /  Please specify the magnitude and direction of the effect size, /  along w...	

What is the size of the effect for research question /  2a: The relationship between referee country /  implicitÂ skin-tone prejudice and red cards /  received by dark skin-tones players?  /  /   /  Please spec...

What is the size of the effect for research question /  2b: The relationship between referee country /  explicit skin-tone /  prejudice and red cards received by dark skin-tones /  players?  /  /   /  Please spe...	
What other steps/analyses did you run that are worth mentioning? / Include effect sizes in a similar format as above if necessary.	

You may use the space below to paste the script you used to run the / analyses. (Optional)