New Data, New Statistics: from Reproducibility Crisis toward New Requirements to Data Analysis and Presentation in Social Sciences

Publication type Article
Status Published
National Research University Higher School of Economics
Institute of Sociology FCTAS RAS
Address: Russian Federation, Moscow
Journal nameSotsiologicheskie issledovaniya
EditionIssue 12

The article analyzes main causes and consequences of the interdisciplinary crisis of the reproducibility and reliability of the results of scientific research that has unfolded in the social sciences in parallel with the «data revolution». This crisis is expressed not only in the growing concern of scientists about the reliability of research results and the possibilities to establish the practices securing the transparency of empirical data and the statistical software used for their analysis, but also in disputes on limitations of the routine approach to significance testing and feasibility of alternatives based on Bayesian approach. Some aspects of the relationship between theory and data-driven methods of searching for patterns in empirical data are briefly discussed in the context of describing a new approach to multimodel analysis aiming at evaluation of model robustness and model uncertainty.

Keywordsreproducibility crisis in social sciences, transparency of data and open data, publication, bias, null hypothesis significance testing and Bayesian approach, model robustness, data-driven approach and sociological theory
AcknowledgmentThe paper is supported by the Russian Science Foundation, grant 17-78-20172
Publication date09.01.2019
Cite   Download pdf To download PDF you should sign in

Price publication: 0

Number of purchasers: 0, views: 2227

Readers community rating: votes 0

1. Deviatko I.F. (2016) From «Virtual Lab» to «Social Telescope»: Metaphors of Theoretical and Methodological Innovations in Online Research. In: Online-research in Russia: Trends and Prospects. Ed. by A.V. Shashkin, I.F. Deviatko, S.G. Davydov. Moscow: Online Market Intelldgence: 19–33. (In Russ.)

2. Deviatko I.F. (1991) TETRAD-methodology: A Completion of Procedural Episteme? Vestnik Akademii nauk SSSR [Herald of the Russian Academy of Sciences]. No. 2: 79–94. (In Russ.)

3. Kitchin R. (2017) Big Data, New Epistemologies and Paradigm Shifts. Sotsiologiya: metodologiya, metody, matematicheskoe modelirovanie (4M) [Sociology: Methodology, Methods, Mathematical Modeling (4M)]. No. 44: 111–152(In Russ.)

4. Schrodt Ph.A. (2016) Seven Deadly Sins of Contemporary Quantitative Political Analysis. Sotsiologiya: metodologiya, metody, matematicheskoe modelirovanie (4M) [Sociology: Methodology, Methods, Mathematical Modeling (4M)]. No. 43: 154–210 (In Russ.)

5. Appelbaum M., Kline R., Nezu A., Cooper H., Mayo-Wilson E., Rao S.M. (2018) Reporting Standards for Quantitative Research in Psychology: The APA Publications and Communications Board Task Force Report. American Psychologist. Vol. 73 (1): 3–25. DOI: 10.1037/amp0000191.

6. de Groot A.D. (1956/2014) The Meaning of «Significance» for Different Types of Research. [Transl. and annot. by E.-J.Wagenmakers et al.] Acta Psychologica. Vol. 148 (May): 188–194.

7. Doucouliagos, C., Stanley T.D. (2013) Are All Economic Facts Greatly Exaggerated? Theory Competition and Selectivity. Journal of Economic Surveys. No. 27(2): 316–39.

8. Gigerenzer G. (2004) Mindless Statistics. The Journal of Socio-Economics. Vol. 33(5): 587–606.

9. Glymour C., Scheines R., Spirtes P. and Kelly K. (1987) Discovering Causal Structure. San Diego, CA: Academic Press.

10. Graham J.H., Özener B. (2016) Fluctuating Asymmetry of Human Populations: A Review. Symmetry. Vol. 8(154): 1–36.

11. Haller H., Kraus S. (2002). Misinterpretations of Significance: A Problem Students Share with Their Teachers? Methods of Psychological Research. No. 7(1): 1–20.

12. Ioannidis J.P.A. (2005) Why Most Published Research Findings Are False. PLoS Med. No. 2(8): e124. DOI: 10.1371/journal.pmed.0020124.

13. Kerr N.L. (1998) HARKing: Hypothesizing after the Results Are Known. Personality and Social Psychology Review. Vol. 2(3): 196–217.

14. Lehrer J. (2010) The Truth Wears off: Is There Something Wrong with the Scientific Method? The New Yorker. December, 13.

15. Munafò M.R., Nosek B.A., Bishop D.V.M., Button K.S., Chambers C.D., Percie Du Sert N., Simonsohn U., Wagenmakers E.J., Ware J.J., Ioannidis J.P.A. (2017) A Manifesto for Reproducible Science. Nature Human Behaviour. Vol. 1(0021). DOI:10.1038/s41562-016-0021.

16. Nelson L., Simmons J.P., Simonsohn U. (2012) Let’s Publish Fewer Papers. Psychological Inquiry. Vol. 23(3): 291–293.

17. Nuijten M.B., Hartgerink C.H., van Assen M.A., Epskamp S., Wicherts J.M. The Prevalence of Statistical Reporting Errors in Psychology (1985–2013). Behavior Research Methods. Vol. 48: 1205–1226.

18. Oakes M. Statistical Inference: A Commentary for the Social and Behavioral Sciences. New York: Wiley, 1986.

19. Open Science Collaboration (Nosek B.A. et al.) (2015) Estimating the Reproducibility of Psychological Science. Science. 2015. No. 349(6251): aac4716. DOI: 10.1126/science.aac4716.

20. Rosenthal R. (1979) The «File Drawer Problem» and Tolerance for Null Results. Psychological Bulletin. Vol. 86(3): 838–641.

21. Rouder J.N., Speckman P.L., Sun D., Morey R.D., Iverson G. (2009) Bayesian t Tests for Accepting and Rejecting the Null Hypothesis. Psychonomic Bulletin & Review. No. 16(2): 225–237. DOI:10.3758/ PBR.16.2.225.

22. Selvin H.C. (1957) A Critique of Tests of Significance in Survey Research. American Sociological Review. Vol. 22(5): 519–527.

23. Stapel D.A., Lindenberg S. (2011) Coping with Chaos: How Disordered Contexts Promote Stereotyping and Discrimination. Science. No. 332(6026): 251–253.

24. Wagenmakers E-J., Verhagen J., Ly A., Bakker M., Lee M.D., Matzke D., Rouder J.N., Morey R.D. (2015). A Power Fallacy. Behavior Research Methods. No. 47(4): 913–917. DOI: 10.3758/s13428-014-0517-4.

25. Yong E. (2012) Replication Studies: Bad Copy. Nature. No. 485(7398): 298–300.

26. Young C. (2018) Model Uncertainty and the Crisis in Science. Socius: Sociological Research for a Dynamic World. Vol. 4: 1–7. DOI: 10.1177/2378023117737206.

Система Orphus