Silberzahn et al. (2018) 領讀摘要

--- title: Silberzahn et al. (2018) 領讀摘要 tags: TCPSR,Reproducibility description: View the slide with "Slide Mode". --- ## Silberzahn et al. (2018) 領讀摘要 --- ## 這項研究的價值  - 心理科學史上第一次群眾外包資料分析專案 - 近期關注度最高的案例：Neuroimage data analysis [(Botvinik-Nezer et al., 2020)](https://www.nature.com/articles/s41586-020-2314-9) - 參考[TCPSR 線上研讀會#6](https://youtu.be/UNcRkVXy9jE) --- ## 啟動這項研究的原因 [總主持人公開召募信](https://docs.google.com/document/d/1uCF5wmbcL90qvrk_J27fWAvDcDNrO9o_APkicwRkOKc/edit) - 一篇論文呈現一種分析方法與結果，實際上只有研究者知道跑了多少種。 - 研究者跑了多種分析，再以主觀挑選符合預期的分析結果，並不符合嚴謹的科學方法論。 - 不同組研究者針對同一個假設，分析同一筆資料，會有多少種分析方法？結果差異有多大？ --- ## 假設與資料來源 - 職業足球比賽皮膚黑的球員會比皮膚白的球員收到較多的紅牌嗎？ - [2012-2013歐洲四國職業足球比賽紀錄](https://osf.io/47tnc/) --- ### 工作階段 |階段|工作目標|學習重點| |:---:|:---|:---| |1|建立資料集|編輯資料編碼簿| |2|招募小組；工作前調查|調查項目相當於預先註冊| |3|第一輪小組分析工作|| |4|小組成果輪流互評|相當於初次正式投稿| |5|第二輪小組分析工作|| |6a|開放討論；總體分析|相當於修改後再提交| |6b|撰寫總報告|| |7|正式投稿期刊；小組外專家評審|| --- ## (預先註冊)小組工作前調查 - 13題必答、可附上示範腳本。 - 瀏覽[問卷項目](https://hackmd.io/_6NPEJXfTZG4oWiYx2KwqQ?view#%E5%B7%A5%E4%BD%9C%E5%89%8D%E8%AA%BF%E6%9F%A5%E5%95%8F%E5%8D%B7) - 33個小組回覆；29個小組完成最後報告。 --- ## 全部分析結果 ![](https://fivethirtyeight.com/wp-content/uploads/2016/03/truth-vigilantes-soccer-calls2.png?w=2048) source: [(Aschwanden, 2016)](https://fivethirtyeight.com/features/failure-is-moving-science-forward/) --- ## 各小組認為資料支持假設的可能性評估 ![](https://journals.sagepub.com/na101/home/literatum/publisher/sage/journals/content/ampa/2018/ampa_1_3/2515245917747646/20181024/images/large/10.1177_2515245917747646-fig4.jpeg) source: [Silberzahn et al. (2018) Figure 4](https://journals.sagepub.com/doi/figure/10.1177/2515245917747646?) --- ## 對QRP的反思 - p-hacking: 看到正面證據的期待 -> 不斷嘗試逼出顯著結果 - 歧路花園: 儘可能做各種可行的分析 -> 執行與假設最一致的分析 - 評審偏好/偏誤: 評審與原作者意見不一致的創新解方？ --- ## 延伸閱讀 > Bishop, D. V. (2020). The psychology of experimental psychologists: Overcoming cognitive constraints to improve research: The 47th Sir Frederic Bartlett Lecture. Quarterly Journal of Experimental Psychology, 73(1), 1–19. doi: 10.1177/1747021819886519 --- ## 衍生問題 1. 今天準備要開始一項新研究，你如何決定分析方法？ 2. 如果分析結果不如預期，你有怎麼樣的處理經驗？ 3. 最後討論到的三個QRP議題，你曾有相關的經驗嗎？ 請在[紀錄共筆](https://hackmd.io/RPooc0rURJiUjWN33j6Qlg)摘要個人與彼此的意見 --- ##### 工作前調查問卷 What transformations (if any) were applied to the variables. Please / be specific. Were any cases excluded, and why? What is the name of the statistical technique that you employed? Please describe the statistical technique you chose in more detail. / Be specific, especially if your choice is not one you consider to / be well-known. What are some references for the statistical technique that you / chose? Which software did you use? If you used multiple kinds, please / indicate what was accomplished with each piece of software (e.g., / Data cleaning - R; Model estimation - SAS) What distribution did you specify for the outcome variable of red / cards? What variables were included as covariates (or control variables) / when testing research question 1: The relationship between player skin tone and red cards / received? What variables were included as covariates (or control variables) / when testing research question 2a: The relationship between referee country implicit skin-tone prejudice and red cards received by... What variables were included as covariates (or control variables) / when testing research question 2b: The relationship between referee country explicit skin-tone prejudice and red cards received by... What theoretical and/or statistical rationale was used for your / choice of covariates included in the models? What unit is your effect size in? What is the size of the effect for research question 1: The relationship between player skin tone and red cards / received? / / Please specify the magnitude and direction of the effect size, / along w... What is the size of the effect for research question / 2a: The relationship between referee country / implicitÂ skin-tone prejudice and red cards / received by dark skin-tones players? / / / Please spec... What is the size of the effect for research question / 2b: The relationship between referee country / explicit skin-tone / prejudice and red cards received by dark skin-tones / players? / / / Please spe... What other steps/analyses did you run that are worth mentioning? / Include effect sizes in a similar format as above if necessary. You may use the space below to paste the script you used to run the / analyses. (Optional)