Project Details
Projekt
Enhancing Reproducibility and Robustness of Observational Social Science Research
Subject Area
Empirical Social Research
Term
since 2021
Project identifier
Deutsche Forschungsgemeinschaft (DFG) - Project number 464507200
The project performs reproducibility and robustness analyses for articles that use the same large-N observational data (European Social Survey data). The articles differ, however, in disciplines, journals, author constellations, and other contextual factors that are likely to be associated with different reproducibility and robustness rates. Our project proceeds in four steps: (A) We started with an openness audit applied to about 1,200 articles (checking the availability of data and code). (B) We perform reproducibility analyses on a random subset of 100 articles with open materials (do the results reproduce when the authors' code is run on their data). (C) Next, we plan correctness/congruence checks for these articles (absence of coding errors, congruence between what is reported in the articles and what is done in the code). (D) Finally, we will perform robustness analyses (do the results hold for seemingly arbitrary changes in data preparation and analysis, such as changes in weighting, imputations, or outlier management). With these four steps, we aim to provide a comprehensive audit of various threats to verifiable research. In contrast to existing research, our audit includes articles from both high-impact and low-impact journals and allows comparisons across disciplines with different adaptation to the FAIR principles (findable, accessible, interoperable, and reusable materials). This will allow the identification of research steps and areas where interventions to promote reproducibility and robustness appear to be most important and effective. The three interrelated, overarching research objectives for the second funding phase are: (1) Completing our analyses of "What" are the rates of articles classified as reproducible and robust through the execution of audits (C) and (D); (2) Analyzing "Why" there is variation, focusing on information at the article, author, and journal level that may indicate key levers for improvement; and (3) Working on "How" to improve reproducibility and robustness through (computational) tools that can be easily used by replicators, authors, and editors.
DFG Programme
Priority Programmes
