如何兼容開放科學與資料科學
===
陳紹慶
慈濟大學人類發展與心理學系
![](http://www.uidownload.com/files/738/714/533/company-identity-logo-new-twitter-icon.png =50x50)@SauChin_Chen
![](https://osf.io/static/img/circle_logo.png =50x50)開放科學中心大使
---
先談談兩個案例
---
# 精挑細選的預測模型
---
<!-- .slide: style="text-align: left;"> -->
[陳昇瑋老師的簡報:網路購書大數據-給出版者的洞察分析](http://www.iis.sinica.edu.tw/~swc/talk/big_data_in_books.html)有三張投影片吸引我的注意:
![cite from](https://i.imgur.com/ju8OFBa.jpg =500x500)
---
<!-- .slide: style="text-align: left;"> -->
這三類暢銷書的預測模型是建立在**暢銷指數**與**首季銷量**的高相關係數,相關係數是一種效果量(effect size),是真實母數的估計值。根據統計學家[Cohen(1988)](http://www.polyu.edu.hk/mm/effectsizefaqs/thresholds_for_interpreting_effect_sizes2.html)的建議,相關係數高於0.7是非常大的效果量。
![effect size benchmark](http://www.polyu.edu.hk/mm/effectsizefaqs/formula/t1.jpg)
---
<!-- .slide: style="text-align: left;"> -->
**如果你是出版社老闆:** 用這些模型企畫下一季要出版的新書,能提高收益嗎?
**如果你是社會科學家:** 這些模型能呈現這個社會的閱讀品味嗎?
---
# 重要性微不足道的臉書實驗
---
<!-- .slide: style="text-align: left;"> -->
2014年中有一份引發網路到媒體熱議好幾天的臉書情緒渲染實驗:
![PANS2014: Facebook experiment](https://i.imgur.com/vkHo2WP.png =600x)
---
<!-- .slide: style="text-align: left;"> -->
實驗結果顯示受測的臉書使用者接受大量負面情緒訊息,個人的臉書訊息負面字眼會顯著增加**0.04%(*P = 0.007*)**。但是仔細看論文,會發現效果量相當微小:
![FB EXP results](https://i.imgur.com/OnmDxmZ.png =800x)
---
![effect size benchmark](http://www.polyu.edu.hk/mm/effectsizefaqs/formula/t1.jpg)
一位臉書使用者的貼文要累積達一萬字,才會出現四個負面字眼。
---
<!-- .slide: style="text-align: left;"> -->
**如果你是馬克.祖克柏:** 值不值得採用這項研究調整臉書的演算法?
**如果你是社會科學家:** 能根據這項結果,給大眾使用臉書的有益建議嗎?
---
我不好奇各位會怎麼回答以上問題。
---
如果能做調查,我最想知道現在的你最想問那一種問題。
---
你自已想問那一種問題,也許反映你的科學觀點。
---
# 經典科學 vs. 資料科學
**有什麼不同?**
---
## 差異一:目的觀
**經典科學:** 根據可重現的事實建立能被考驗的理論。
**資料科學:** 找出能有效預測、控制現象發生的模型。
---
## 差異二:知識論
**經典科學:** 有事前計畫,約定資料收集程序,收集處理格式化資料。
**資料科學:** 不一定有收集資料程序,待格式化或已格式化資料皆是處理對象。
---
![Two sciences?](https://i.imgur.com/wC4YDyp.jpg)
[可參考這一串臉書討論](https://www.facebook.com/groups/853552931365745/permalink/1523498554371176/)
---
經典科學與資料科學都有失敗的案例
---
科學家要如何辨識與修正導致失敗的問題
---
# 資料科學失敗案例
---
## google預測流感失敗的案例
---
- 2008
![Ginsberg et al.(2008) Nature](https://i.imgur.com/emXZWDa.png =800x)
---
- 2011
![Cook et al.(2011) PLOS One](https://i.imgur.com/hrWCld4.png =800x)
---
- 2014
![Lazer et al. (2014) Science](https://i.imgur.com/uVrHOm6.png =800x)
---
![Bye Bye](https://i.imgur.com/ijuxnNa.png)
[大數據分析的迷思:以谷歌流感趨勢預測為例](https://scitechvista.nat.gov.tw/c/HqLz.htm)
---
## google預測流感失敗的教訓
**只依賴資料與統計模型,無法萃練真正有效的預測模型。**
---
## Meehl(1990) 因子大雜繪(Crud Factors)
[**Crud Factors**](http://goodsciencebadscience.nl/?p=471)
*the phenomenon that ultimately everything correlates to some extent with everything else.*
---
- Lkyyn & Meehl(1966)
- a questionnaire to 57,000 high school seniors
- cross-tabulated a total of 15 variables
- 105 correlations
- 101 (96%) of 105 correlations are statsitically significant ( $p = 10^{-6}$ )
---
# 經典科學失敗案例
---
## 無力的權力姿勢(Underpowered Power Posing)
---
![2 min gesture change your life](https://i-chentsai.innovarad.tw/wp-content/uploads/2016/09/amycuddy.jpg =800x)
---
![](https://static01.nyt.com/images/2017/10/22/magazine/22cover/22cover-superJumbo.jpg =600x)
### [When the Revolution Came for Amy Cuddy](https://www.nytimes.com/2017/10/18/magazine/when-the-revolution-came-for-amy-cuddy.html)
---
![power pose original paper](https://i.imgur.com/AYePHCy.png =800x)
---
![Dana Carney](http://media.npr.org/assets/img/2016/09/30/carney-d7f1dde4110eb4fdd6ef04321ad87906522605a7-s300-c85.jpg =200x)
![Dana Carney Statement](https://i.imgur.com/m9NuEyS.png =1000x)
---
![2015 Replication](http://www.slate.com/content/dam/RISK-GRAPH.jpg.CROP.promovar-mediumlarge.jpg =1000x)
---
![Power pose p curve](https://whyevolutionistrue.files.wordpress.com/2015/05/screen-shot-2015-05-10-at-6-26-47-am.png =800x)
([Simmons & Simonsohn, 2015](http://datacolada.org/37))
---
![Power pose testosteron](http://anabolicmen.com/wp-content/uploads/2014/01/body-language-and-testosterone-levels.jpg =1000x)
(Carney et al, 2010)
---
![Testosteron reanalysis from Gary McClelland](https://i.imgur.com/TBhdsnQ.jpg =800x)
**It's gender! No matter pose!**
---
### Exploration-as-Confirmation Fallacy
德州神槍手謬誤(Texas sharpshooter fallacy)
![](https://vivifychangecatalyst.files.wordpress.com/2016/04/texas.jpg)
---
- 只分析符合研究者觀點的資料,或只採用突顯有利證據的分析。
- 分析結果也許符合**預測**,其實無意或刻意丟失有意義的資料。
---
![Meta analysis on 6 studies, without knowledge of power pose](https://i.imgur.com/BOm5iqX.png =800x)
(Gronau et al., 2017)
---
<!-- .slide: style="text-align: left;"> -->
Andrew Gleman: ["... the original study is too noisy to be useful."](http://andrewgelman.com/2017/09/27/somewhat-agreement-fritz-strack-regarding-replications/)
More story: [When the Revolution Came for Amy Cuddy](https://www.nytimes.com/2017/10/18/magazine/when-the-revolution-came-for-amy-cuddy.html) & a lot of comments
Amy Cuddy strike back: [Reply to Simmons and Simonsohn
](https://s3.us-east-2.amazonaws.com/amy.cuddy.com.website/ssrn_3054952.pdf)
---
# 修正失敗案例要面對的挑戰
---
**一門科學存在過半的陽性結果,通常存在高比例的偽陽性研究**
---
![What is false positive](http://www.timvanderzee.com/inhoud/uploads/2016/07/type-i-and-type-ii-errors.jpg =800x)
---
![Many sciences have high proportion of positive results](http://journals.plos.org/plosone/article/figure/image?size=large&id=10.1371/journal.pone.0010068.g001 =600x)
---
![Why most published research findings are false](https://i.imgur.com/lYz4kSE.png =900x)
估計一門科學存在偽陽性結果比例的數據模擬方法
---
### 要解釋清楚得講很久,還好有[視覺化網頁](http://shinyapps.org/showapp.php?app=http://lmpp10e-mucesm.srv.mwn.de:3838/felix/PPV&by=Michael%20Zehetleitner%20and%20Felix%20Sch%C3%B6nbrodt&title=When%20does%20a%20significant%20p-value%20indicate%20a%20true%20effect?&shorttitle=When%20does%20a%20significant%20p-value%20indicate%20a%20true%20effect?)。
---
- 過半陽性結果,高比例偽陽性的參數:
- Prior = 0.5
- alpha = .05
- Power = .33
- p-hacking= .85
---
![High false positive](https://i.imgur.com/LOzotje.png =800x)
---
### 真實的狀況比數據模擬更嚴重
![OSC 2015](https://pbs.twimg.com/media/CNewoLtUYAAzayr.png =800x)
---
### 為什麼有經典科學這麼多偽陽性的結果?
![Bias and Flexibity](https://cdn.cos.io/media/images/Hypothetico-deductive_scientific_method-1.original.png =800x)
---
- 偏誤(Bias): 顯著結果易被接受;期刊編輯與經費提供者偏好原創性研究
- 彈性(Flexibity): 不擇手段逼出著結果;忽視造成低考驗力的條件
---
## 開放科學的解方
- 嚴謹(Rigor):研究預先註冊,降低導致偏誤的各種條件
- 透明(Transparency):可追溯獲取資料的紀錄,讓研究彈性公開於陽光下
---
# 資料科學也要克服偽陽性
---
- 過度適應(Overfitting): 符合訓練資料的模型無法符合測試資料
![Overfitting](https://upload.wikimedia.org/wikipedia/commons/thumb/1/19/Overfitting.svg/1200px-Overfitting.svg.png =600x)
---
<!-- .slide: style="text-align: left;"> -->
**只用訓練資料(一項不完全公開原始資料的研究)建立的模型(理論),成功通過測試資料的機率極低(缺乏直接再現的研究)。**
---
# 開放科學的解決方案
- 增加新模型/新發現被重製的機會 = 累積**有效**的再現結果
- 2011年之後的大規模再現專案: Many Labs [1](http://econtent.hogrefe.com/doi/pdf/10.1027/1864-9335/a000178),[2](https://osf.io/8cd4r/wiki/home/),[3](http://curatescience.org/logos/pdf-icon.gif),[4](https://osf.io/zc4nv/),[5](https://docs.google.com/document/d/1tnPqr2JSpODQjJ8yB-lC6sCwuWoJn0kWEfgg8mZH5nY/edit); [Reproducibility Project: Psychology](http://etiennelebel.com/documents/osc(2015,science).pdf); [The XPhi Replicability Project](https://sites.google.com/site/thexphireplicabilityproject/home) ...
---
<!-- .slide: style="text-align: left;"> -->
- 大規模再現專案的共同點:原創研究發表後,長期缺乏有系統的直接再現。
- 有潛力但乏人問津、有文化差異但缺乏證據的研究,能不能運用這種模式確認或翻新現在的知識?
---
<!-- .slide: style="text-align: left;"> -->
![](https://psysciacc.files.wordpress.com/2017/10/cropped-cropped-psa.png =30x)**[The Psychological Science Accelerator: A Distributed Laboratory Network心理科學加速器](https://christopherchartier.com/2017/09/21/the-psychological-science-accelerator-a-distributed-laboratory-network/)**
我為眾人貢獻研究、眾人與我進行研究
---
## PSA任務 (from AMPPS Draft)
- "... to accelerate the accumulation of reliable and generalizable evidence in psychological science, reducing the distance between truth about human behavior and mental processes and our current understanding."
---
## PSA方針 (from AMPPS Draft)
- "... we attempt to meet this challenge with a **distributed laboratory network** that is ongoing (as opposed to time or task limited), diverse (both in terms of human subjects and participating researchers), and inclusive (we welcome ideas, contributions, study proposals, or other input from anyone in the field of psychology)."
---
## 如何讓全世界一起參與我的研究
![Selection Process](https://i.imgur.com/GmnsfTp.png =400x)
---
## 三個月內的成長
![Increasing Member Lab](https://christopherchartierblog.files.wordpress.com/2017/10/10-3-map1.png =1000x)
---
## PSA第一件合作專案
[The generalizability of the valence-dominance model of face social perception.](http://www.sciencemag.org/news/2017/11/new-accelerator-aims-bring-big-science-psychology)
![](https://paw.princeton.edu/sites/default/files/images/content/mind-faces.jpg =500x)
---
## PSA的優勢
1. For science: non-WEIRD countries, can ask new questions that leverage the variation created by a distributed network, studies are pre-registered, no publication bias.
2. For scientists: fast data collection, access to more participants, requires clarity on study procedure/hypotheses, opportunity to collaborate, opportunity to learn new skills (e.g., pre-registering, how to report methods transparently), opportunity to learn about new paradigms, stimuli.
---
## 我看PSA的挑戰
- Collaborative data collection strategies: web-based vs. lab-based
- Collaborative data analysis strategies: Unified vs. multiverse?
- Funding issues.
- Transfer research culture from lab to lab, from society to society.
---
## 整合科學([meta-science](https://en.wikipedia.org/wiki/Metascience_(research)))即將興起?
如果資料科學與經典科學的目的觀與知識論不必然互斥,兩方結合也許是一種整合科學的樣態。
---
# 結語
- 科學家認清從事的科學目的:探索?確證?
- 學習彼此的長處
- 經典/開放科學:精進可被事實考驗的理論
- 資料科學:清晰、更容易上手的資料分析流程
{"metaMigratedAt":"2023-06-14T14:22:16.936Z","metaMigratedFrom":"YAML","title":"如何兼容開放科學與資料科學","breaks":"true","slideOptions":"{\"transition\":\"slide\",\"theme\":\"white\",\"center\":true,\"width\":\"80%\",\"height\":\"100%\",\"margin\":0,\"minScale\":1,\"maxScale\":1}","contributors":"[]"}