Project Details
Projekt
The reproducibility and robustness of secondary analyses in educational research: The role of publication bias and researcher degrees of freedom
Applicants
Dr. Malte Jansen; Dr. Aleksander Kocaj
Subject Area
Developmental and Educational Psychology
General and Domain-Specific Teaching and Learning
General and Domain-Specific Teaching and Learning
Term
since 2021
Project identifier
Deutsche Forschungsgemeinschaft (DFG) - Project number 464313518
In recent years, the robustness of results of empirical studies in psychology, the social sciences, medicine and related disciplines has been called into question. Simmons, Nelson and Simonsohn (2011, p. 1359) showed that “researcher degrees of freedom” in data analysis may contribute strongly to non-replicable findings. Researcher degrees of freedom are not only related to data analysis, however, but also affect the dissemination of the results. For example, researchers tend to decide against the submission of null results for publication (Rosenthal, 1979).In the proposed project, we aim to estimate the robustness of results estimates of secondary data analyses in the field of educational research. We assume that the robustness will vary based on (a) the decisions for a particular dataset (or a number of datasets) to study the effect, (b) the decisions made when preparing and analyzing the dataset and (c) the decision whether to write up a manuscript and which results to include in that manuscript (possibly affected by publication bias and selective reporting). Our project aims to disentangle and empirically describe these effects. We will first analyze applications for secondary data analyses by researchers at the research data center at the Institute for Educational Quality Improvement (FDZ at IQB). In the data applications, researchers describe their central questions and hypotheses as well as their planned analytic approach including data sets, central variables, and statistical analyses. We will analyze around 570 data applications from over 900 researchers as well as around 164 publications resulting from those applications to construct a direct measure of publication bias in educational research (Aim A1). By comparing the research questions described in the data applications and the results described in the publications, we will also gain insights into selective reporting practices (Aim A2) The second two aims relate to the reproducibility and robustness of results. We aim to, first, examine the reproducibility of the published results of a selected subsample of data applications—that is, we aim to independently reproduce the same analysis based on the same dataset (Aim B1). Second, besides the direct reproduction of published data applications, we will also examine how researchers’ decision on a specific dataset and analytic strategy affects study results and how robust these results are (Aim B2). To study robustness, we combine methods of Integrative Data Analysis (IDA) to test if results and conclusions generalize to different data sets and (2) methods of Specification Curve Analysis (SCA) to test how different analytic strategies within one data set affect the results. Overall, the project helps to gain insights into different sources of result heterogeneity that might affect the reproducibility and robustness of secondary analyses in educational research.
DFG Programme
Priority Programmes
